*
AI2Bot
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Diffbot
DuckAssistBot
FacebookBot
Google-Extended
GPTBot
Kangaroo Bot
Meta-ExternalAgent
Meta-ExternalFetcher
OAI-SearchBot
omgili
PanguBot
PerplexityBot
Timpibot
Webzio-Extended
YouBot


* GPTBot Google-Extended anthropic-ai CCBot Omgilibot cohere-ai ChatGPT-User OAI-SearchBot PerplexityBot Amazonbot YouBot Claude-Web Claudebot Claude-User Claude-SearchBot Diffbot Bytespider ImagesiftBot Meltwater Seekr


* Amazonbot anthropic-ai Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai DataForSeoBot GPTBot Google-Extended magpie-crawler omgili omgilibot PetalBot PerplexityBot SemrushBot weborama-fetcher


* AdsBot-Google Mediapartners-Google AmazonAdBot


AmazonAdBot * Bingbot msnbot Slurp FacebookBot proximic Verity/1.1 SirdataBot


* Claude-User Claude-SearchBot ChatGPT-User OAI-SearchBot Perplexity-User Applebot-Extended Amazonbot anthropic-ai GPTBot ClaudeBot cohere-ai meta-externalagent


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent Googlebot DuckDuckBot Twitterbot Bingbot Facebot


* facebookexternalhit LinkedInBot Twitterbot Baiduspider GPTBot CCBot Google-Extended


* AdsBot-Google Mediapartners-Google AdsBot-Google-Mobile


dotbot AhrefsBot SemrushBot SemrushBot-SA SemrushBot-BA *


Twitterbot FacebookBot Elastic-Crawler *


*
meta-externalagent
Meta-ExternalFetcher
ChatGPT-User
OAI-SearchBot
Bytespider
DuckAssistBot
Perplexity-User
Timpibot
ClaudeBot
Amazonbot
GPTBot
PerplexityBot
CCBot
FacebookBot
Ai2Bot-Dolma
Claude-SearchBot
Claude-User
omgilibot
TtoYouBot
WebGPTBot
Verity/1.1
Scope3/2.0 (scope3.com)
AmazonAdBot


Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Mozilla/5.0 (compatible; Baiduspider/3.0; +http://www.baidu.com/search/spider.html)

Mozilla/5.0 (compatible; Baiduspider/4.0; +http://www.baidu.com/search/spider.html)

Mozilla/5.0 (compatible; Baiduspider/5.0; +http://www.baidu.com/search/spider.html)

Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)


Googlebot AdsBot-Google grapeshot * Mediapartners-Google ias_crawler GPTBot AI2Bot Ai2Bot-Dolma ClaudeBot Bytespider cohere-training-data-crawler DeepSeekBot meta-externalagent PetalBot


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


* Googlebot-News Amazonbot anthropic-ai Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai Diffbot FacebookBot Google-Extended GPTBot magpie-crawler omgili omgilibot PerplexityBot Google-CloudVertexBot meta-externalagent meta-externalfetcher


GPTBot ChatGPT-User ClaudeBot Google-Extended Applebot-Extended anthropic-ai cohere-ai CCBot ICC-Crawler Bytespider OAI-SearchBot PerplexityBot Meta-ExternalAgent


008 UnisterBot JikeSpider Zealbot WebStripper WebCopier Fetch Teleport HTTrack magpie-crawler MJ12bot DotBot spbot BLEXBot BUbiNG Yandex Baiduspider OAI-SearchBot ChatGPT-User GPTBot ClaudeBot Meta-ExternalAgent PerplexityBot Google-Extended AmazonAdBot * Twitterbot facebookexternalhit


* Googlebot Googlebot-News Googlebot-Image Googlebot-Video Google-InspectionTool Google-NotebookLM bingbot BingPreview msnbot msnbot-media Google-Extended facebookexternalhit GPTBot SemrushBot AhrefsBot


Mediapartners-Google AdsBot-Google AdsBot-Google-Mobile AmazonAdBot Googlebot-Image CCBot ChatGPT-User GPTBot Google-Extended PerplexityBot ClaudeBot dotbot Riddler BLEXBot AhrefsBot MauiBot *


Mozilla/5.0 (compatible; Pinterestbot/1.0; +https://www.pinterest.com/bot.html)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Pinterestbot/1.0; +https://www.pinterest.com/bot.html)


* Yandex AdsBot-Google facebookexternalhit Twitterbot


Mediapartners-Google*
TruliaBot
Twitterbot
*
Amazonbot
Applebot
GPTBot
A6-Indexer
AhrefsBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot-Extended
Baiduspider
betaBot
BLEXBot
botify
brands-bot-logo
Brightbot
Bytespider
calculon
CCBot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
collection@infegy.com
Cotoyogi
Crawl4AI
CrunchBot
deepcrawl
DeepSeekBot
Diffbot
DirBuster
DTS
DuckAssistBot
EasouSpider
FacebookBot
FirecrawlAgent
FriendlyCrawler
GlutenCrawler
Google-CloudVertexBot
Google-Extended
GrapeshotCrawler
iaskspider
ICC-Crawler
ImagesiftBot
img2dataset
JackCrawl
Jetslide
linkapedia_bot
Mediatoolkitbot
MegaIndex
meta-externalagent
meta-externalfetcher
Microsoft BITS
Microsoft URL Control
Microsoft-WebDAV-MiniRedir
MistralAI-User
news-please
Novellum
NTENTbot
Nutch
omgili
Omgilibot
OpenLinkProfiler
Perplexity-User
PerplexityBot
PetalBot
Primalbot
python-requests
Re-Animator
Riddler
SafeDNSBot
SBIntuitionsBot
Scrapy
SearchAtlas
SemanticScholarBot
SemrushBot
SentiBot
Sogou
Spider-Pig
sqlmap
StorygizeBot
Test Certificate Info
TikTokSpider
Timpibot
Turnitin
uipbot
vBSEO


Bingbot * Google-HotelAdsVerifier/2.0 Google-HotelAdsVerifier AdIdxBot


Yeti
Baiduspider
Baiduspider-image
Baiduspider-video
Baiduspider-news
Baiduspider-favo
Baiduspider-cpro
Baiduspider-ads
Sogou web spider
Sogou inst spider
Sogou spider
Sogou wap spider
360Spider
360Spider-Image
360Spider-Video
PetalBot
*
AdsBot-Google


Googlebot-News * Yahoo-MMCrawler psbot BaiduImagespider BaiduMobaider TAGGY Mail.RU_Bot grapeshot


* dcbs-google thunderstonesa Googlebot bingbot


anthropic-ai
AmazonBot
Claude
Claude-Web
claudeweb
Amzn-SearchBot
Amzn-User
AndiBot
cohere-ai
ClaudeBot
ChatGPT-User
Googlebot-News
GPTBot
CCBot
Google-Extended
DataForSeoBot
Diffbot
Diffbot-user
Kagibot
FacebookBot
magpie-crawler
NewsNow
news-please
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Scrapy
TurnitinBot
Applebot-Extended
Meta-ExternalAgent
meta-externalagent
OAI-SearchBot
Bytespider
Claude-User
Claude-SearchBot
AI2Bot
AhrefsBot
Ai2Bot-Dolma
Applebot
Arquivo-web-crawler
AspiegelBot
AwarioRssBot
AwarioSmartBot
Barkrowler
coccocbot-web
cohere-training-data-crawler
crawler4j
Crawlspace
dotbot
DuckAssistBot
FriendlyCrawler
Google-CloudVertexBot
https://hada.news
https://www.imediaethics.org
iaskspider/2.0
ICC-Crawler
icc-crawler
ICC Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
LivelapBot
MauiBot
Meltwater
Meta-ExternalFetcher
MJ12bot
Mojeek
MojeekBot
MoodleBot
NeticleBot
Netvibes
PanguBot
PetalBot
SeekportBot
Seekr
SemrushBot
SeznamBot
Sidetrade Indexer Bot
Timpibot
Turnitin
VelenPublicWebCrawler
Webzio-Extended
yacy
yacybot
Yandex
YandexAdditional
YandexAdditionalBot
YandexBot
YandexImages
YandexNews
Baiduspider
Baiduspider-image
BaiduSpider-render
Baiduspider-video
Sogou Web Spider
yisouspider


* AdsBot-Google-Mobile AdsBot-Google PetalBot dotbot AhrefsBot


* Googlebot Googlebot-Image Mediapartners-Google Adsbot-Google


*
008 Disallow: /
AddThis.com Disallow: /
admantx Disallow: /
AhrefsBot Disallow: /
BDCbot Disallow: /
Bender Disallow: /
BIXOCRAWLER Disallow: /
bl.uk_lddc_bot Disallow: /
BLEXBot Disallow: /
BUbiNG Disallow: /
Cliqzbot Disallow: /
CNCDialer Disallow: /
crawler4j Disallow: /
CrystalSemanticsBot Disallow: /
CyberAlert Disallow: /
DigExt Disallow: /
discobot Disallow: /
discoverybot Disallow: /
dloader Disallow: /
dloader(NaverRobot) Disallow: /
DOC Disallow: /
dotbot Disallow: /
Download Ninja Disallow: /
DTS Agent Disallow: /
Exabot Disallow: /
Ezooms Disallow: /
FairShare Disallow: /
Fetch Disallow: /
Flamingo_SearchEngine Disallow: /
Genieo Disallow: /
Gigabot Disallow: /
grub-client Disallow: /
Heritrix Disallow: /
heritrix/3.3.0 Disallow: /
HTTrack Disallow: /
ia_archiver Disallow: /
integromedb Disallow: /
IstellaBot Disallow: /
JikeSpider Disallow: /
Jyxobot Disallow: /
k2spider Disallow: /
Kimengi Disallow: /
Kimengi/nineconnections.com Disallow: /
larbin Disallow: /
LexxeBot/1.0 Disallow: /
libwww Disallow: /
linko Disallow: /
Livelapbot Disallow: /
magpie-crawler Disallow: /
Maxthon Disallow: /
MetaURI Disallow: /
Microsoft.URL.Control Disallow: /
MJ12bot Disallow: /
Moreover Disallow: /
Moreoverbot Disallow: /
MSIECrawler Disallow: /
nabot Disallow: /
NaverBot Disallow: /
NerdByNature.Bot Disallow: /
netEstate NE Crawler Disallow: /
NetSeer crawler Disallow: /
Newscan Disallow: /
NextGenSearchBot Disallow: /
NPBot Disallow: /
nutch Disallow: /
Offline Explorer Disallow: /
omgilibot Disallow: /
Orthogaffe Disallow: /
PetalBot Disallow: /
PiplBot Disallow: /
Pixray-Seeker Disallow: /
proximic Disallow: /
psbot Disallow: /
QuerySeekerSpider Disallow: /
rogerbot Disallow: /
seokicks Disallow: /
SEOkicks-Robot Disallow: /
SiteBot Disallow: /
SiteBot/0.1 Disallow: /
sitecheck.internetseer.com Disallow: /
SiteSnagger Disallow: /
Slurp Disallow: /
sogou Disallow: /
Sosospider Disallow: /
spbot Disallow: /
Spinn3r Disallow: /
Teleport Disallow: /
TeleportPro Disallow: /
trendictionbot Disallow: /
trovitBot Disallow: /
TurnitinBot Disallow: /
UbiCrawler Disallow: /
uMBot-LN Disallow: /
UnisterBot Disallow: /
UniversalFeedParser Disallow: /
WBSearchBot Disallow: /
WebCopier Disallow: /
WebReaper Disallow: /
WebStripper Disallow: /


* GPTBot ia_archiver Googlebot Applebot adidxbot


*
Twitterbot
facebookexternalhit
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
Amazonbot
AmazonBuyForMe
anthropic-ai
Applebot-Extended
bigsur.ai
Bytespider
CCBot
ChatGLM
ChatGLM-Spider
ChatGPT-Operator
ChatGPT-User
ClaudeBot
Claude-User
Claude-Web
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
DataForSeoBot
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
gemini-deep-research
GoogleAgent-Mariner
Google-Extended
Google-NotebookLM
GPTBot
ICC-Crawler
ImagesiftBot
imageSpider
img2dataset
Kangaroo Bot
KlaviyoAIBot
laion-huggingface-processor
LCC
LinerBot
Manus-User
meta-externalagent
meta-externalfetcher
MistralAI-User
netEstate
netEstate Imprint Crawler
NovaAct
omgili
OmigiliBot
PanguBot
Perplexity-User
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
SemrushBot-SWA
sider.ai
Spider
TavilyBot
TheKnowledgeAI
TimpiBot
TwinAgent
VelenPublicWebCrawler
webzio-extended
YouBot
AhrefsBot
SearchmetricsBot
rogerbot
deepcrawl
OnCrawl
Screaming Frog SEO Spider
MJ12bot
Sistrix
barkrowler


* Mediapartners-Google Amazonbot anthropic-ai Bytespider CCBot ChatGPT-User Claude-Web Google-Extended GPTBot


* GPTBot ChatGPT-User Google-Extended ClaudeBot PerplexityBot Bytespider Applebot-Extended


* Googlebot GPTBot ClaudeBot CCBot Bytespider Google-Extended meta-externalagent Amazonbot Applebot-Extended PerplexityBot ChatGPT-User OAI-SearchBot anthropic-ai Claude-Web cohere-ai Diffbot FacebookBot PetalBot Timpibot img2dataset omgili Scrapy DeepSeekBot chatgpt-operator claude-searchbot GrokBot MistralAI-User TikTokSpider AI2Bot PanguBot Amzn-SearchBot


* AhrefsBot SemrushBot CCBot meta-externalagent


Amazonbot anthropic-ai AhrefsBot PerplexityBot CCBot Google-Extended GPTBot ChatGPT-User *


* Googlebot Googlebot-Mobile Twitterbot facebookexternalhit SiteAuditBot


Amazonbot
Applebot
GPTBot
A6-Indexer
AhrefsBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot-Extended
Baiduspider
betaBot
BLEXBot
botify
brands-bot-logo
Brightbot
Bytespider
calculon
CCBot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
collection@infegy.com
Cotoyogi
Crawl4AI
CrunchBot
deepcrawl
DeepSeekBot
Diffbot
DirBuster
DTS
DuckAssistBot
EasouSpider
FacebookBot
FirecrawlAgent
FriendlyCrawler
GlutenCrawler
Google-CloudVertexBot
Google-Extended
GrapeshotCrawler
iaskspider
ICC-Crawler
ImagesiftBot
img2dataset
JackCrawl
Jetslide
linkapedia_bot
Mediatoolkitbot
MegaIndex
meta-externalagent
meta-externalfetcher
Microsoft BITS
Microsoft URL Control
Microsoft-WebDAV-MiniRedir
MistralAI-User
news-please
Novellum
NTENTbot
Nutch
omgili
Omgilibot
OpenLinkProfiler
Perplexity-User
PerplexityBot
PetalBot
Primalbot
python-requests
Re-Animator
Riddler
SafeDNSBot
SBIntuitionsBot
Scrapy
SearchAtlas
SemanticScholarBot
SemrushBot
SentiBot
Sogou
Spider-Pig
sqlmap
StorygizeBot
Test Certificate Info
TikTokSpider
Timpibot
Turnitin
uipbot
vBSEO
VegeBot
VelenPublicWebCrawler
Web Link Validator
WebCruiser


GPTBot ByteSpider ToutiaoSpider koubeispider koubei.com amap.com *


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


*
Amazonbot
Applebot-Extended
Bytespider
CCBot
ClaudeBot
CloudflareBrowserRenderingCrawler
Google-Extended
GPTBot
meta-externalagent
MJ12bot
Mediapartners-Google
IsraBot
Orthogaffe
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
fast
grub-client
k2spider
NPBot
WebReaper
Omgilibot
ChatGPT-User
openai-crawler
anthropic-ai
Claude-Web
PerplexityBot
YouBot
Applebot
facebookexternalhit
MetaAI-Bot
HuggingFaceBot
Scrapy
python-requests
curl
Wget


* Amazonbot anthropic-ai AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai DataForSeoBot Diffbot FacebookBot Google-Extended GPTBot magpie-crawler NewsNow news-please omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot Scrapy TurnitinBot


*
AddSearchBot
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Andibot
anthropic-ai
Applebot
Applebot-Extended
Awario
bedrockbot
bigsur.ai
Brightbot 1.0
Bytespider
CCBot
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
Google-Firebase
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
LinerBot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
meta-webindexer
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
ShapBot
Sidetrade indexer bot
TerraCotta
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YaK
YandexAdditional
YandexAdditionalBot
YouBot


ai2bot-dolma anthropic-ai applebot-extended bytespider ccbot claude-user claude-web claudebot cohere-ai diffbot duckassistbot friendlycrawler omgili omgilibot perplexity-user perplexitybot scrapy timpibot youbot webzio-extended kangaroo-bot google-extended AmazonAdBot *


* Google-Extended AwarioRssBot AwarioSmartBot Baiduspider DataForSeoBot Diffbot ImagesiftBot magpie-crawler NewsNow news-please omgili omgilibot peer39_crawler peer39_crawler/1.0 Scrapy TurnitinBot


*
Googlebot
AddSearchBot
AI2Bot
anthropic-ai
Applebot-Extended
BedrockBot
bigsur.ai
Brightbot 1.0
Bytespider
CCBot
ChatGPT Agent
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
Cohere-Training-Data-Crawler
Cotoyogi
Datenbank Crawler
DeepSeek
DeepSeek-Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
EchoboxBot
EtaoSpider
FacebookBot
FireCrawlAgent
GoogleAgent-Mariner
GoogleAgent-Shopping
GPTBot
ICC-Crawler
ImagesiftBot
img2dataset
Kangaroo Bot
Kraken
LinerBot
Linkfluence Yak Bot
Meta-ExternalAgent
Meta-ExternalFetcher
MistralAI-Crawler
MyCentralAIScraperBot
netEstate Imprint Crawler
news-please
NovaAct
ObjectsSearch
omgili
omgilibot
Operator
PanguBot
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
Quora-Bot
quillbot.com
ResearchBot
SBIntuitionsBot
scrapy
SeekrBot
Sidetrade indexer bot
TaraGroup Intelligent Bot
ThinkBot
thinkers-bot
Timpibot
Webzio-Extended
wpbot
YouBot
facebookexternalhit
AmazonAdBot
NanoInteractive
Opebot-v (https://www.1plusx.com)


* LinkedInBot Twitterbot facebookexternalhit EchoboxBot ClaudeBot CCBot Google-CloudVertexBot Google-Extended Meta-ExternalAgent OAI-SearchBot ChatGPT-User GPTBot PerplexityBot Perplexity-User


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* anthropic-ai cohere-ai omgili omgilibot Applebot-Extended Bytespider CCBot ChatGPT-User ChatGPT DeepSeek DeepSeekBot GPTBot OAI-SearchBot ClaudeBot Claude-Web Diffbot FacebookBot Google-Extended Meta-ExternalAgent Meta-ExternalFetcher PerplexityBot Timpibot Webzio-Extended YouBot cXensebot


* OAI-SearchBot ChatGPT-User PerplexityBot AndiBot ClaudeBot Claude-User Claude-SearchBot


* Mediapartners-Google yahoo-slurp msnbot grapeshot


Alexa
Amazon-Kendra
AmazonBot
ByteDance
ByteSpider
CC-Crawler
CCBot
ChatGLM
ClearScopeBot
CommonCrawl
DataForSEOBot
Diffbot
Falcon
FlyRiver
Goose
Grammarly
Iaskspider
imagesiftBot
Img2dataset
PetalBot
Quillbot
Scrapy
Sidetrade
Sonic
TikTokSpider
Timpibot
VelenPublicWebCrawler
Swiftbot
Elastic-Crawler Elastic Cloud
Elastic-Crawler
Elastic Cloud
*


AdsBot-Google AdsBot-Google-Mobile Mediapartners-Google AdIdxBot HTTrack wget XoviBot AhrefsBot AhrefsSiteAudit Riddler FDM ExtLinksBot SemrushBot SemrushBot-SA MJ12bot QuetextBot BLEXBot grapeshot BluechipBacklinks barkrowler GPTBot ClaudeBot Claude-Web cohere-ai DataForSeoBot PetalBot meta-externalagent AwarioRssBot AwarioSmartBot Amazonbot ChatGPT-User IbouBot YouBot *


* TTD-Content TinEye "TinEye" CCBot ChatGPT-User GPTBot Google-Extended anthropic-ai ClaudeBot Omgilibot Omgili Diffbot Bytespider ImagesiftBot cohere-ai PerplexityBot FacebookBot Amazonbot YouBot Applebot-Extended meta-externalagent


Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/w4mwnpbXf3MFAbxOkJRw; +http://www.echobox.com)

Mozilla/5.0 (compatible; NewsRoom.BI/0.1; +http://www.newsroom.bi/bot.html)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36(KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; mrfCompass-Marshall/1.0)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Jukebox/1.0)


anthropic-ai
AmazonBot
Claude
Claude-Web
claudeweb
Amzn-SearchBot
Amzn-User
AndiBot
cohere-ai
ClaudeBot
ChatGPT-User
Googlebot-News
GPTBot
CCBot
Google-Extended
DataForSeoBot
Diffbot
Diffbot-user
Kagibot
FacebookBot
magpie-crawler
NewsNow
news-please
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Scrapy
TurnitinBot
Applebot-Extended
Meta-ExternalAgent
meta-externalagent
OAI-SearchBot
Bytespider
Claude-User
Claude-SearchBot
AI2Bot
AhrefsBot
Ai2Bot-Dolma
Applebot
Arquivo-web-crawler
AspiegelBot
AwarioRssBot
AwarioSmartBot
Barkrowler
coccocbot-web
cohere-training-data-crawler
crawler4j
Crawlspace
dotbot
DuckAssistBot
FriendlyCrawler
Google-CloudVertexBot
https://hada.news
https://www.imediaethics.org
iaskspider/2.0
ICC-Crawler
icc-crawler
ICC Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
LivelapBot
MauiBot
Meltwater
Meta-ExternalFetcher
MJ12bot
Mojeek
MojeekBot
MoodleBot
NeticleBot
Netvibes
PanguBot
PetalBot
SeekportBot
Seekr
SemrushBot
SeznamBot
Sidetrade Indexer Bot
Timpibot
Turnitin
VelenPublicWebCrawler
Webzio-Extended
yacy
yacybot
Yandex
YandexAdditional
YandexAdditionalBot
YandexBot
YandexImages
YandexNews
Baiduspider
Baiduspider-image
BaiduSpider-render
Baiduspider-video
Sogou Web Spider
yisouspider


*
Mediapartners-Google
Meltwater
archive-fr.com
360Spider
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
Baiduspider-image
Baiduspider-video
Brightbot 1.0
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Crawlspace
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
Meta-ExternalAgent
Meta-ExternalFetcher
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
PetalBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
Sogou Spider
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YisouSpider
YouBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* CriteoBot/0.1 GPTBot CCBot Applebot-Extended AmazonAdBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent Googlebot-image


AmazonAdBot Google-InspectionTool ChatGPT-User GPTBot OAI-SearchBot *


Google-Extended GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Adsbot-Google PetalBot YandexBot Baiduspider Sogou SEOkicks-Robot MegaIndex.ru BLEXBot *


*
Twitterbot
bingbot
AhrefsbBot
SemrushBot
BLEXBot
MJ12bot
AmazonAdBot
Pinterestbot
trendictionbot
NTENTbot
DotBot
rogerbot
Exabot
GetIntent Crawler
Yandex
YandexMobileBot
PhantomJS
SputnikBot
SeznamBot
Cliqzbot
Mail.RU_Bot
Baiduspider
Baiduspider-video
Baiduspider-image
SentiBot
MauiBot (crawler.feedback+wc@gmail.com)
Mappy
WotBot


* AdsBot-Google\ APIs-Google AdsBot-Google-Mobile Mediapartners-Google


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* GoogleBot GoogleBot-Mobile Mediapartners-Google Yandex eSobiSubscriber Exabot MJ12bot SemrushBot SemrushBot-SA


* grapeshot GPTBot ClaudeBot Google-Extended Amazonbot CCBot


* FacebookBot facebookexternalhit Twitterbot/1.0 msnbot yaanibot


* magpie-crawler CCBot omgili omgilibot Claude-Web ClaudeBot anthropic-ai cohere-ai Bytespider PetalBot Scrapy Applebot-Extended GPTBot ChatGPT-User PerplexityBot Perplexity-User Google-CloudVertexBot OAI-SearchBot YandexAdditional YandexAdditionalBot


* Mediapartners-Google Amazonbot anthropic-ai Bytespider CCBot ChatGPT-User Claude-Web Google-Extended GPTBot


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


Crescent Internet ToolPak HTTP OLE Control v.1.0


* GPTBot OAI-SearchBot Claude-SearchBot Claude-User ChatGPT-User PerplexityBot


Mediapartners-Google*
Googlebot-Image
googlebot
Twitterbot
msnbot
Slurp
AliyunSecBot
Amazonbot
anthropic-ai
Applebot-Extended
archive.org_bot
AudigentAdBot
AwarioRssBot
AwarioSmartBot
BLEXBot
Bytespider
CCBot
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
cohere-ai
DataForSeoBot
Diffbot
DuckAssistBot
EchoboxBot
FacebookBot
FirecrawlAgent
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
GPTBot
ImagesiftBot
Jetslide
magpie-crawler
Meta-ExternalAgent
meta-externalagent
Meta-ExternalFetcher
meta-externalfetcher
MyCentralAIScraperBot
NewsNow
news-please
OAI-SearchBot
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Perplexity-User
Poseidon Research Crawler
quillbot.com
Quora-Bot
Scrapy
SeekrBot
SeznamHomepageCrawler
TaraGroup Intelligent Bot
Timpibot
TurnitinBot
ViennaTinyBot
YouBot
*


AI2Bot Ai2Bot-Dolma aiHitBot Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler DuckAssistBot FacebookBot Google-Extended GoogleOther GoogleOther-Image GoogleOther-Video GPTBot img2dataset Meta-ExternalAgent MyCentralAIScraperBot omgili omgilibot Quora-Bot TikTokSpider YouBot AdsBot-Google AdsBot-Google-Mobile AdsBot-Google-Mobile-Apps *


* Googlebot Googlebot-image Google-Extended ChatGPT-User PerplexityBot CCBot


SemrushBot
SemanticScholarBot
PetalBot
Baiduspider
Yandex
trendictionbot
WinHTTP Example/1.0
ClaudeBot
anthropic-ai
CCBot
ChatGPT-User
GPTBot
*
RIT Storm Crawler
Googlebot
google
SiteimproveBot
SiteimproveBot-Crawler
AHC/1.0
AHC/2.0
AHC/2.1


* Yandex AdsBot-Google facebookexternalhit Twitterbot


applebot-extended
bytespider
ccbot
claudebot
google-extended
cohere-training-data-crawler
deepseekbot
deepseek
deepseek-crawler
diffbot
facebookbot
meta-externalagent
gptbot
grokbot
img2dataset
laiondownloader
mistralai-crawler
omgili
omgilibot
pangubot
Teleport
Webwhacker
Webzip
Net Attache
SiteSnagger
HTTrack
WebCapture
WebSauger
*


Amazonbot
Applebot-Extended
Brightbot
ClaudeBot
Google-Extended
GPTBot
PerplexityBot
PetalBot
Scrapy
uptimerobot
viberbot
YaK
Yandex
Yeti
Applebot
Bingbot
Discordbot
DuckDuckBot
facebookexternalhit
Googlebot
Googlebot-Image
Googlebot-Video
Googlebot-News
Google-InspectionTool
LinkedInBot
msnbot
Pinterestbot
Screaming Frog SEO Spider
seznambot
Slurp
teoma
TelegramBot
Twitterbot
*


ClaudeBot ImagesiftBot SenutoBot CCBot AudigentAdBot *


*
Mediapartners-Google
5emeRue
5erue
adequat
adequat-systems
AmiSoftware
Argus
Ask n read
asknread.com
Augure
auramundi
Bloodhound
CCBot
ChatGPT-User
Cision
Claudebot
Cliqzbot
coexel
ConveraCrawler
Corporama
cydralspider
Digimind
Dotbot
Download Ninja
downloadexpress
EDD
ellisphere
eureka
Europresse
Explore
Factiva
Fasterfox
Fetch
FirstRain
gammaSpider
GnowitNewsbot
Google-Extended
GPTBot
grub-client
ia_archiver
ia_archiver-web.archive.org
indexer
infoseek
Jetbot
k2spider
Kantar
kbcrawl
Knowings
larbin
leadbox
libwww
linkfluence
linko
manageo
MaxPointCrawler
mediacompil
Meltawer
Meltwater
mention
Moreover
MSIECrawler
mytwip
newscan-online
NewsNow
Newzbin
NPBot
Nutch
ObjectsSearch
omgili
omgilibot
opinion-tracker
Pimptrain
proxem
QuepasaCreep
Qwam content intelligence
Raven
readability.com
Riddler
rogerbot
scoop.it
score3
SearchmetricsBot
Sindup
sitecheck.internetseer.com
SiteSnagger
spotter
Synthesio
Talkwater
Teleport
TeleportPro
trendeo
trendybuzz
TunitinBot
TurnitinBot
up2news
vecteurplus
Verif
verticalsearch
vsw


* Yandex Mail.Ru Freshbot IstellaBot AhrefsBot trendictionbot sentibot grapeshot heritrix MJ12bot um-IC SiteAuditBot Slurp dotbot SeekportBot barkrowler


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Googlebot Bingbot YandexBot DuckDuckBot Qwantify Yeti Sogou MojeekBot ChatGPT-User ChatGPT-User/2.0 OAI-SearchBot PerplexityBot Claude-Web Claude-SearchBot Claude-User DuckAssistBot Amzn-SearchBot ManusBot facebookexternalhit FacebookBot Twitterbot LinkedInBot Slackbot Applebot GPTBot Google-Extended GoogleOther Google-CloudVertexBot anthropic-ai ClaudeBot CCBot Meta-ExternalAgent Amazonbot Applebot-Extended cohere-ai Bytespider Diffbot AI2Bot MistralAI-User Omgilibot webzio-extended YouBot TerracottaBot


* grapeshot GPTBot ClaudeBot Google-Extended Amazonbot CCBot


*
MJ12bot
infometrics-bot
AI2Bot
Ai2Bot-Dolma
Bytespider
CCBot
Diffbot
FriendlyCrawler
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
omgili
omgilibot
Scrapy
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended


MJ12bot AhrefsBot Applebot SeznamBot DotBot SemrushBot


*
MegaIndex.ru
Barkrowler
aranhabot
SEOkicks
DotBot
BLEXBot
MJ12bot
DataForSeoBot
FemtosearchBot
serpstatbot
WebPageTest.org
crawler_faq
GeedoBot
crawler_eb_germany
MetaJobBot
CCBot
VelenPublicWebCrawler
hgfAlphaXCrawl
FullStoryBot
NE Crawler


* GPTBot CCBot PerplexityBot Perplexity-User OAI-SearchBot ChatGPT-User


Yandex Mail.Ru Rambler Googlebot AdsBot-Google Googlebot-Image Google-Read-Aloud FeedFetcher-Google GoogleDocs GoogleOther *


* Nuclei WikiDo Riddler PetalBot Zoominfobot Go-http-client Node/simplecrawler CazoodleBot dotbot/1.0 Gigabot Barkrowler BLEXBot magpie-crawler


anthropic-ai CCBot ChatGPT-User Claude-Web ClaudeBot GPTBot Google-Extended PerplexityBot *


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent grapeshot


*
ChatGPT-user
CCBot
GPTBot
Google-Extended
ImagesiftBot
FacebookBot
Omgilibot
Amazonbot
anthropic-ai
Claude-Web
cohere-ai
omgili
PerplexityBot
YouBot
Bytespider
Diffbot
Applebot-Extended
OAI-SearchBot
ClaudeBot
Google-CloudVertexBot
Meta-ExternalAgent
Meta-ExternalFetcher
Quora-Bot
DataForSeoBot
Webzio-Extended
FriendlyCrawler
ChatGPT-User
meta-externalagent
meta-externalfetcher
TeleportPro
Qwam content intelligence
Timpibot
Kangaroo Bot
AI2Bot
VelenPublicWebCrawler
img2dataset
Sidetrade indexer bot
PanguBot
ISSCyberRiskCrawler
Ai2Bot-Dolma
Claude-SearchBot
Claude-User
MistralAI-User
DeepSeekBot
Perplexity-User


*
Ai2Bot-Dolma
AI2Bot
anthropic-ai
Applebot-Extended
Arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
Barkrowler
Bytespider
CCBot
ClaudeBot
cohere-ai
cohere-training-data-crawler
crawler4j
dotbot
Diffbot
FacebookBot
FriendlyCrawler
GPTBot
https://hada.news
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MJ12bot
PanguBot
Scrapy
Seekr
Timpibot
VelenPublicWebCrawler
Webzio-Extended


* Yandex AdsBot-Google facebookexternalhit Twitterbot


Mediapartners-Google AmazonAdBot Baiduspider coccoc grapeshot *


* YandexAccessibilityBot YandexCalendar YandexDialogs YandexDirect YandexDirectDyn YandexFavicons YaDirectFetcher YandexMobileBot YandexMarket YandexMetrika YandexMobileScreenShotBot YandexOntoDBAPI YandexPartner YandexRCA YandexRenderResourcesBot YandexSearchShop YandexTracker YandexUserproxy YandexVideoParser YandexScreenshotBot YandexAdditionalBot


* Googlebot-Image Mediapartners-Google Adsbot-Google Twitterbot


* ia_archiver Googlebot-Image Googlebot-News Mediapartners-Google psbot


* Googlebot-Image zs-crawler zoho-scrapy


* GPTBot ChatGPT-User OAI-SearchBot Google-Extended ClaudeBot Claude-SearchBot PerplexityBot SlySearch SmartDownload SpankBot spanner SuperBot SupperHTTP Surfbot ia_archiver


Vast_Bot
YandexBot
Tag Inspector Spider
*
AdsBot-Google
GPTBot
OAI-SearchBot
ChatGPT-User
Amazonbot
CCBot
Claude-User
Claude-SearchBot
DuckAssistBot
PerplexityBot
Perplexity-User
GrokBot
AhrefsBot
Google-InspectionTool


Ultraseek
Lucidworks-Anda/2.0/0.8; +;
Lucidworks-Anda/2.0/0.10; +;
*
googlebot


Alexabot * Googlebot AdsBot-Google Mediapartners-Google APIs-Google GPTBot PerplexityBot ClaudeBot


* applebot-extended bytespider ccbot claudebot google-extended cohere-training-data-crawler deepseekbot deepseek deepseek-crawler diffbot facebookbot meta-externalagent gptbot grokbot img2dataset laiondownloader mistralai-crawler omgili omgilibot pangubot


* YandexBot Googlebot AdsBot-Google AdsBot-Google-Mobile


* Mediapartners-Google* Pinterestbot dotbot AhrefsBot DataForSeoBot SemrushBot


* GPTBot OAI-SearchBot ChatGPT-User ClaudeBot PerplexityBot anthropic-ai CCBot Bytespider


TelegramBot Yandex * trendkite-akashic-crawler PetalBot


yandex
bing
yahoo
PetalBot
CriteoBot/0.1
bytespider
byte spider
bytedance
OAI-SearchBot
Applebot
ClaudeBot
GPTBot
Amazonbot
ChatGPT-User
CCBot
PerplexityBot
AmazonAdBot
*


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


OAI – SearchBot
OAI-SearchBot
ChatGPT-User
ChatGPT-User/2.0
GPTBot
anthropic-ai
ClaudeBot
claude-web
Claude-User
PerplexityBot
Perplexity-User
Google-Extended
Gemini-Deep-Research
BingBot
Amazonbot
Applebot
Applebot-Extended
FacebookBot
meta-externalagent
LinkedInBot
Bytespider
DuckAssistBot
cohere-ai
AI2Bot
CCBot
Diffbot
omgili
TimpiBot
YouBot
*


AdsBot-Google-Mobile AdsBot-Google * Googlebot-Image


AhrefsBot
AhrefsSiteAudit
YepBot
SemrushBot
SemrushBot-SA
Rogerbot
Exabot
barkrowler
MJ12bot
rogerbot
Dotbot
dotbot
DataForSeoBot
ia_archiver
magpie-crawler
MBCrawler
iubenda-radar
Gigabot
BlackWidow
ChinaClaw
Custo
DISCo
Download\ Demon
eCatch
EirGrabber
EmailSiphon
EmailWolf
Express\ WebPictures
ExtractorPro
EyeNetIE
FlashGet
GetRight
GetWeb!
Go!Zilla
Go-Ahead-Got-It
GrabNet
Grafula
HMView
HTTrack
Image\ Stripper
Image\ Sucker
Indy\ Library
InterGET
Internet\ Ninja
JetCar
JOC\ Web\ Spider
larbin
LeechFTP
Mass\ Downloader
MIDown\ tool
Mister\ PiX
Navroad
NearSite
NetAnts
NetSpider
Net\ Vampire
NetZIP
Octopus
Offline\ Explorer
Offline\ Navigator
PageGrabber
Papa\ Foto
pavuk
pcBrowser
RealDownload
ReGet
SiteSnagger
SmartDownload
SuperBot
SuperHTTP
Surfbot
tAkeOut
Teleport\ Pro
VoidEYE
Web\ Image\ Collector
Web\ Sucker
WebAuto
WebCopier
WebFetch
WebGo\ IS
WebLeacher
WebReaper
WebSauger
Website\ eXtractor
Website\ Quester
WebStripper
WebWhacker
WebZIP
Wget
Widow
WWWOFFLE
Xaldon\ WebSpider
Zeus


* AdsBot-Google-Mobile AdsBot-Google PetalBot dotbot AhrefsBot


*
Twitterbot
facebookexternalhit
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
Amazonbot
AmazonBuyForMe
anthropic-ai
Applebot-Extended
bigsur.ai
Bytespider
CCBot
ChatGLM
ChatGLM-Spider
ChatGPT-Operator
ChatGPT-User
ClaudeBot
Claude-User
Claude-Web
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
DataForSeoBot
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
gemini-deep-research
GoogleAgent-Mariner
Google-Extended
Google-NotebookLM
GPTBot
ICC-Crawler
ImagesiftBot
imageSpider
img2dataset
Kangaroo Bot
KlaviyoAIBot
laion-huggingface-processor
LCC
LinerBot
Manus-User
meta-externalagent
meta-externalfetcher
MistralAI-User
netEstate
netEstate Imprint Crawler
NovaAct
omgili
OmigiliBot
PanguBot
Perplexity-User
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
SemrushBot-SWA
sider.ai
Spider
TavilyBot
TheKnowledgeAI
TimpiBot
TwinAgent
VelenPublicWebCrawler
webzio-extended
YouBot
AhrefsBot
SearchmetricsBot
rogerbot
deepcrawl
OnCrawl
Screaming Frog SEO Spider
MJ12bot
Sistrix
barkrowler


* semetrical GPTBot OAI-SearchBot Applebot-Extended PerplexityBot Perplexity-ai CCBot YouBot ClaudeBot anthropic-ai Claude-Web grapeshot bingbot AmazonAdBot ozone Meta-ExternalAgent


* facebookexternalhit Googlebot bingbot msnbot AhrefsBot baiduspider sogou dotbot GoogleOther magpie-crawler weborama-fetcher ias_crawler 360Spider GPTBot CCBot anthropic-ai Amazonbot Claude-Web ClaudeBot PetalBot


Crescent Internet ToolPak HTTP OLE Control v.1.0


bingbot Pinterest AhrefsBot AspiegelBot CCbot *


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent ia_archiver


* Googlebot Googlebot-Mobile Mediapartners-Google Googlebot-Image Googlebot-News Google-Extended GPTBot PerplexityBot Amazonbot meta-externalagent APIs-Google CCBot ClaudeBot OAI-SearchBot


yaanibot grapeshot * OAI-SearchBot GPTBot


Mozilla/4.0 (compatible; BullsEye; Windows 95)


* AdsBot-Google AdsBot-Google-Mobile Mediapartners-Google Googlebot Googlebot-Image GPTBot ChatGPT-User Claude-Web PerplexityBot GoogleOther OAI-SearchBot ClaudeBot


* ADmantX AlphaBot Amazonbot anthropic-ai AwarioRssBot AwarioSmartBot BLEXBot Brightbot Buzzbot Bytespider CCBot ChatGPT-User claritybot Claude-Web ClaudeBot cohere-ai Diffbot FacebookBot FriendlyCrawler Google-Extended GPTBot huggingface ImagesiftBot img2dataset magpie-crawler Meltwater Neevabot news-please NewsNow Nutch OAI-SearchBot omgili omgilibot http://panscient.com Perplexity-ai PerplexityBot PetalBot PiplBot http://scoop.it Scrapy Seekr SentiBot SeznamBot TurnitinBot YouBot ZumBot


Mediapartners-Google GoogleBot-Image Twitterbot grapeshot * Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot Claude-SearchBot Google-Extended GPTBot meta-externalagent PerplexityBot OAI-SearchBot ia_archiver archive.org_bot ia_archiver-web.archive.org


Mail.RU_Bot MJ12bot ExaBot CrazyWebCrawler-Spider SemrushBot AhrefsBot BLEXBot Yandex *


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


*
008
AddThis.com
admantx
BDCbot
Bender
bl.uk_lddc_bot
BLEXBot
BUbiNG
CNCDialer
CrystalSemanticsBot
CyberAlert
discobot
discoverybot
DOC
DotBot
Download Ninja
Exabot
Ezooms
Fetch
Flamingo_SearchEngine
grub-client
heritrix
HTTrack
integromedb
IstellaBot
JikeSpider
Jyxobot
k2spider
larbin
LexxeBot/1.0
libwww
linko
magpie-crawler
Maxthon
MJ12bot
Moreoverbot
MSIECrawler
NerdByNature.Bot
netEstate NE Crawler
NetSeer crawler
Newscan
NextGenSearchBot
NPBot
Offline Explorer
Pixray-Seeker
psbot
rogerbot
seokicks
SiteBot
SiteBot/0.1
sitecheck.internetseer.com
SiteSnagger
sogou
Sosospider
spbot
Spinn3r
Teleport
TeleportPro
trovitBot
TurnitinBot
UbiCrawler
UnisterBot
WBSearchBot
WebCopier
WebReaper
WebStripper
WebZIP
WeSEE:Search
wget
wotbox
Yasni
Zao
Zealbot
ZyBORG
anthropic-ai
AI2Bot
Ai2Bot-Dolma
AmazonBot
ApifyBot
ApifyWebsiteContentCrawler
Applebot-Extended
Bytespider
CCBot
ChatGLM-Spider
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Diffbot
FacebookBot
FirecrawlAgent
GPTBot
iAskBot
iaskspider
ICC-Crawler
Kangaroo Bot


* Yandex Googlebot Googlebot-News YandexNews


msnbot
*
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot-Extended
CCBot
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-Web
Claude-User
cohere-ai
Diffbot
discobot
dotbot
FacebookBot
google-extended
Google-Extended
Googlebot
Googlebot-Image
GPTBot
omgilibot
PerplexityBot
TurnitinBot
TtoYouBot
magpie-crawler
Slurp
yacybot
meta-externalagent
Meta-ExternalFetcher
OAI-SearchBot
Bytespider
DuckAssistBot
Perplexity-User
Timpibot
WebGPTBot
Scope3/2.0 (scope3.com)
AmazonAdBot


*
ChatGPT-User
Google-CloudVertexBot
Google-Extended
GPTBot
OAI-SearchBot
AhrefsBot
AI2Bot
Amazonbot
anthropic-ai
Applebot-Extended
Arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
Bytespider
CCBot
Claude-Web
ClaudeBot
Claude-SearchBot
Claude-User
coccocbot-web
cohere-ai
cohere-training-data-crawler
Crawlspace
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
Gensparkbot
https://hada.news
ICC-Crawler
ImagesiftBot
https://www.imediaethics.org
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
magpie-crawler
Meta-ExternalAgent
Meta-ExternalFetcher
Mojeek
MojeekBot
MoodleBot
Netvibes
NewsNow
omgili
omgilibot
PanguBot
PerplexityBot
Perplexity-ai
Perplexity-User
PetalBot
Scrapy
Seekr
SemrushBot
SeznamBot
Timpibot
Turnitin
TurnitinBot
VelenPublicWebCrawler
Webzio-Extended
yacy
yacybot
Yandex
YandexAdditional
YandexAdditionalBot
YouBot


AmazonAdBot Google-InspectionTool ChatGPT-User GPTBot OAI-SearchBot Google-Extended *


*
GPTBot
ClaudeBot
Google-Extended
Meta-ExternalAgent
Applebot-Extended
DeepSeekBot
Grok
GrokBot
Amazonbot
CCBot
cohere-ai
sitebot
Slurp
CazoodleBot
Cazoodlebot
sistrix
MJ12bot
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper
psbot
Exabot
Speedy
dotbot
Bloglines/3.1
Jyxobot/1
cityreview
WijuBot


* AI2Bot Amazonbot Applebot-Extended anthropic-ai Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai Diffbot DuckAssistBot FacebookBot Google-Extended GPTBot Meta-ExternalAgent Meta-ExternalFetcher OAI-SearchBot PerplexityBot


* Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai DataForSeoBot GPTBot Google-Extended magpie-crawler omgili omgilibot PetalBot PerplexityBot SemrushBot Scrapy TurnitinBot weborama-fetcher


* GPTBot Google-Extended Applebot-Extended


* Google-Extended GPTBot ChatGPT-User CCBot ClaudeBot Bytespider Amazonbot Applebot-Extended ia_archiver Arquivo-web-crawler anthropic-ai OAI-SearchBot PerplexityBot Claude-Web


Googlebot Googlebot-Image AdsBot-Google GoogleOther *


* AhrefsBot AhrefsSiteAudit AwarioBot BLEXBot Bytespider CriteoBot DataForSeoBot DotBot GrapeshotCrawler grapeshot MJ12bot MegaIndex MojeekBot SemrushBot YandexBot Yeti YisouSpider


* Yandex AdsBot-Google facebookexternalhit Twitterbot


Twitterbot * Mediapartners-Google bingbot PetalBot Google-Extended


CCBot ChatGPT-User GPTBot Google-Extended Google-CloudVertexBot Omgilibot Omgili FacebookBot ClaudeBot Claude-SearchBot Claude-User Diffbot DuckAssistBot PerplexityBot Perplexity-User cohere-ai cohere-training-data-crawler Meta-ExternalAgent Meta-ExternalFetcher Meta-WebIndexer Timpibot Webzio-Extended YouBot Amazonbot Bytespider anthropic-ai OAI-SearchBot VelenPublicWebCrawler Gemini-Deep-Research ICC-Crawler MistralAI-User *


* OAI-SearchBot ChatGPT-User PerplexityBot FirecrawlAgent AndiBot ExaBot PhindBot YouBot GPTBot CCBot Google-Extended Googlebot Bingbot


* OAI-SearchBot GPTBot PerplexityBot CazoodleBot dotbot Gigabot


* AmazonAdBot anthropic-ai Applebot-Extended GPTBot Google-Extended ClaudeBot Claude-Web Perplexity-User PerplexityBot OAI-SearchBot Bytespider


* Fasterfox Teleport TeleportPro wget BlackWidow


* GPTBot CCBot Bytespider PerplexityBot Perplexity-User OAI-SearchBot ChatGPT-User


008 UnisterBot JikeSpider Zealbot WebStripper WebCopier Fetch Teleport HTTrack magpie-crawler MJ12bot DotBot spbot BLEXBot BUbiNG Yandex Baiduspider OAI-SearchBot ChatGPT-User GPTBot ClaudeBot Meta-ExternalAgent PerplexityBot Google-Extended Googlebot * Twitterbot facebookexternalhit AmazonAdBot


Amazonbot anthropic-ai Bytespider CCBot ChatGPT-User Claude-Web FacebookBot Google-Extended GoogleOther GoogleOther-Image GoogleOther-Video GPTBot PerplexityBot cohere-ai omgili omgilibot Diffbot ImagesiftBot Applebot-Extended dotbot rogerbot


AccompanyBot
AwarioRssBot
AwarioSmartBot
AhrefsBot
BLEXBot
BLP_bbot
CCBot
chimebot
MJ12bot
Screaming Frog SEO Spider
Semanticbot
SemrushBot
startmebot
*


bingbot
*
Feedly
Twitterbot
Mediapartners-Google
Google-Display-Ads-Bot
AdsBot-Google-Mobile
AdsBot-Google
CriteoBot
Slurp
Megalodon
ia_archiver
AhrefsBot
SemrushBot
MJ12bot
BLEXBot
GPTBot
ChatGPT-User
Applebot-Extended
CCBot
ImagesiftBot
ClaudeBot
claude-web
PerplexityBot
Meta-ExternalAgent
Meta-WebIndexer
DuckAssistBot
DiffBot
cohere-ai
OAI-SearchBot
anthropic-ai
AI2Bot
Riddler
MLBot
JUST-CRAWLER
DuckDuckBot
Sogou web spider
Rogerbot
Omgilibot
Omgili
peer39_crawler
Bytespider
ias-or
Scrapy
Copilot
Google-NotebookLM
Diffbot
PanguBot
MistralAI-User
EtaoSpider
Claude-SearchBot


Applebot BingBot BingPreview ChatGPT-User Claude-SearchBot Claude-User DuckDuckBot facebookexternalhit Grok-DeepSearch Googlebot GoogleOther LinkedInBot meta-externalfetcher meta-webindexer MistralAI-User OAI-SearchBot PerplexityBot Perplexity-User Slackbot Twitterbot xAI-Grok AI2Bot AI2Bot-Dolma AliyunSecBot Amazonbot anthropic-ai Applebot-Extended bigsur.ai Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler DeepseekBot Diffbot FacebookBot GPTBot GrokBot iAskBot ImagesiftBot meta-externalagent omgili omgilibot peer39_crawler peer39_crawler/1.0 Petalbot Timpibot webzio webzio-extended Googlebot-Image Googlebot-Video *


* FacebookExternalHit Twitterbot Pinterestbot LinkedInBot TikTokSpider Storebot-Google Google-InspectionTool AmazonAdBot Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended Amazonbot meta-webindexer meta-externalagent meta-externalfetcher ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User cohere-training-data-crawler cohere-ai CCBot PanguBot PetalBot Bytespider Diffbot DuckAssistBot MistralAI-User Timpibot Webzio Webzio-Extended archive.org_bot ia_archiver ia_archiver-web.archive.org heritrix


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Googlebot Bingbot BingPreview Pinterestbot facebookexternalhit Applebot OSZKbot upday Mediapartners-Google


* GPTBot OAI-SearchBot ChatGPT-User ClaudeBot PerplexityBot Google-Extended


Slurp
Mail.Ru
bingbot
msnbot
AhrefsBot
discobot
Ezooms
MJ12bot
Flamingo_SearchEngine
magpie-crawler
Rootle Crawler
pimeyes.com crawler
weborama-fetcher
ias-or
Yandex
YandexImages
*


*
Amazonbot
AlibabaBot
Ai2Bot-Dolma
Applebot-Extended
Bytespider
CCBot
ChatGLM-Spider
ClaudeBot
Claude-Web
CloudVertexBot
cohere-training-data-crawler
Cotoyogi
Datenbank Crawler
Diffbot
FacebookBot
Google-Extended
GoogleOther
GPTBot
ICC-Crawler
imageSpider
Kangaroo Bot
laion-huggingface-processor
LCC
meta-externalagent
netEstate Imprint Crawler
omgili
PanguBot
PerplexityBot
SBIntuitionsBot
Spider
Timpibot
VelenPublicWebCrawler
webzio-extended
YouBot
008
Dataprovider.com
dcrawl
HelloworkJobPostingBot
HTTrack
HTTrack 3.0
KrawlerBot
MetaInspector
MetaJobBot
newspaper
Nutch
Offline Explorer
OpenindexSpider
Potions
Scrapy
ServerHunterSpider
StatsDroneBot
CloudflareBrowserRenderingCrawler
aa
AhrefsBot
Attracta
Barkrowler
BrightEdge Crawler
BLEXBot
CaliberBot
Caliperbot
ClarityBot
Clearscopebot
Cloudtrellis
cludo.com
Cocolyzebot
cognitiveSEO Crawler
contentking
Convermax
Cxense
DataForSeoBot
DataForSEO Bot
DomainStatsBot
DotBot
Dragonbot
Huckabot
Huckabuy Bot
hypestat
iaskspider
img2dataset
James BOT
LinkCheck by Siteimprove
LinkChecker Bot
linkchecker.pro
linkdexbot
LinksIndexerBot
MarketGoo
MBCrawler
MegaIndex.ru
MJ12bot
Moz dotbot
Moz rogerbot
Nitro-
NitroBot
online-webceo-bot
Prerender
Pro Sitemaps
Readable
RevvimGort
rogerbot-crawler


*
Almaden
ASPSeek
Axmo
booch
DTS Agent
Downloader
EmailCollector
EmailSiphon
EmailWolf
Expired Domain Sleuth
Franklin Locator
Gaisbot
grub
HughCrawler
iaea.org
lcabotAccept
IconSurf
Iltrovatore-Setaccio
Indy Library
IUPUI
Kittiecentral
larbin
lwp-trivial
MetaTagRobot
Missigua Locator
NetResearchServer
NextGenSearch
NPbot
Nutch
ObjectsSearch
Oracle Ultra Search
PEERbot
PictureOfInternet
PlantyNet
QuepasaCreep
ScSpider
SOFT411
spider.acont.de
Sqworm
SSM Agent
TAMU
TheUsefulbot
TurnitinBot
Tutorial Crawler
TutorGig
WebCopier
WebZIP
ZipppBot
Wotbox
Wget
mozDex
ZoominfoBot
spyglassbot
SeznamBot
Nimbostratus-Bot
DigExt
Cincraw
BLEXBot


* GPTBot ChatGPT-User PerplexityBot ClaudeBot Google-Extended Amazonbot Applebot-Extended CCBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* GPTBot ChatGPT-User ClaudeBot Claude-Web anthropic-ai PerplexityBot TurnitinBot cohere-ai OAI-SearchBot QuillBot


* ia_archiver archive.org_bot heritrix My-heritrix-crawler Timpibot omgili Bytespider DuckAssistBot Applebot-Extended Google-Extended Google-CloudVertexBot GoogleOther GPTBot CCBot Amazonbot FacebookBot cohere-ai cohere-training-data-crawler Omgilibot ClaudeBot Diffbot PanguBot MistralAI-User webz.io DeepSeekBot AmazonAdBot Amzn-SearchBot Amzn-User Claude-SearchBot Claude-User PetalBot Meta-ExternalFetcher Meta-WebIndexer OAI-SearchBot ChatGPT-User PerplexityBot Perplexity-User


Mediapartners-Google
Googlebot
Googlebot-Image
Googlebot-Mobile
Googlebot-News
Googlebot-Video
Adsbot-Google
Twitterbot
Applebot
Bingbot
SiteAuditBot
Publication-Access-for-Facebook
facebookexternalhit
Flipboard
FlipboardProxy
upday
AhrefsBot
AhrefsSiteAudit
5emeRue
5erue
adequat
adequat-systems
AmiSoftware
AwarioRssBot
AwarioSmartBot
Argus
Ask n read
asknread.com
Augure
auramundi
Bloodhound
Cision
coexel
ConveraCrawler
Corporama
cydralspider
Digimind
Download Ninja
downloadexpress
EDD
ellisphere
eureka
Europresse
Explore
Factiva
Fasterfox
Fetch
gammaSpider
grub-client
HTTrack
ia_archiver
ia_archiver-web.archive.org
indexer
infoseek
Jetbot
k2spider
Kantar
kbcrawl
Knowings
larbin
leadbox
libwww
linkfluence
linko
manageo
mediacompil
Meltwater
mention
Moreover
MSIECrawler
mytwip
newscan-online
NewsNow
Newzbin
NPBot
ObjectsSearch
Offline Explorer
opinion-tracker
Pimptrain
proxem
QuepasaCreep
Qwam content intelligence
Raven
readability.com
scoop.it
score3
Sindup
sitecheck.internetseer.com
SiteSnagger
spotter
Synthesio
Talkwater
Teleport
TeleportPro
trendeo
trendybuzz
TunitinBot
TurnitinBot
up2news
vecteurplus


Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/w4mwnpbXf3MFAbxOkJRw; +http://www.echobox.com)

Mozilla/5.0 (compatible; NewsRoom.BI/0.1; +http://www.newsroom.bi/bot.html)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36(KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; mrfCompass-Marshall/1.0)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Jukebox/1.0)


DuckDuckBot Mediapartners-Google Googlebot Googlebot-Image Googlebot-Mobile GoogleProducer Googlebot-Video AdsBot-Google Googlebot_Nauxeo Qwantify Qwant-news VoilaBot msnbot Slurp bingbot Twitterbot facebookexternalhit Applebot Bingbot Facebot grapeshot Flipboard FlipboardProxy weborama-fetcher FeedFetcher-Google ias_crawler ias-sg ias-au ias-jp ias-ie ias-or ias-va ias-sync admantx-apacas admantx-euaspb admantx-usaspb admantx-ussy01 admantx-ussy02 admantx-ussy03 admantx-ussy04 admantx-eusy01 admantx-eusyncbatch01 admantx-usbatch admantx-sap aday-rediffusion PanoramaPQR SeedtagBot AmazonAdBot IAB-Tech-Lab TTD-Content Googlebot-News *


"008"
AddSearchBot
AI2Bot
Ai2Bot-DeepResearchEval
Ai2Bot-Dolma
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-extended
atlassian-bot
AutoRAG
AwarioSmartBot
AzureAI-SearchBot
Big Sur AI
bigsur.ai
Brandwatch
Bravebot
Brightbot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DDM-DCipher*
DDM-DCipher/1.0.7
DDM*
DeepSeekBot
Devin
Diffbot
Direqt Anomura
DuckAssistBot
Echobot Bot
ExaBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
GPTBot
HTTrack
iAsk
iAskBot
iaskspider
ICC Crawler
ICC-Crawler
ImageSiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
Kunato
laion-huggingface-processor
LCC
LINER Bot
LinerBot
LinkupBot
Manus-User
meta-externalagent
meta-externalfetcher
meta-webindexer
mistral.ai
MistralAI-User
netEstate Imprint Crawler
NovaAct
Novellum AI Crawl
Nutch
OAI-SearchBot
Offline Explorer
omgili
opencode
PanguBot
PeopleInc-DCipher-Scraper/1.1.0
Perplexity-User
PerplexityBot
PetalBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
Scrapy
SeekrBot
SemrushBot-OCOB
SemrushBotSwa
ShapBot
Spider
TavilyBot
TerraCotta
Timpibot
Trae
TwinAgent


* AdsBot-Google-Mobile AdsBot-Google Bytespider Meta-ExternalAgent OAI-SearchBot GPTBot CCBot PerplexityBot ClaudeBot Google-Extended Bingbot


*
Amazonbot
Anchor Browser
Bytespider
CCBot
ClaudeBot
FacebookBot
Google-CloudVertexBot
GPTBot
meta-externalagent
Novellum
PetalBot
ProRataInc
Timpibot
FriendlyCrawler


EtaoSpider psbot NPBot-1/2.0 NPBot Googlebot AdsBot-Google-Mobile AdsBot-Google Bingbot AdldxBot *


AwarioBot
Barkrowler
BUbiNG
ContentKing
DotBot
ecoresearchCrawler
MJ12bot
SeznamBot
Vegi bot
Bytedance
Bytespider
CCBot
AhrefsBot
SemrushBot
SemrushBot-BA
*


* AdIdxBot MSNBot Yandex Baiduspider Bingbot


* Twitterbot GPTBot ChatGPT-User CCBot anthropic-ai Amazonbot Applebot-Extended Bytespider ClaudeBot Claude-Web cohere-ai Diffbot Google-Extended Meta-ExternalFetcher PerplexityBot YouBot DeepSeekBot MistralAI-User MistralAI-User/1.0 PetalBot Timpibot YandexAdditional YandexAdditionalBot YandexBot YandexRenderBot Perplexity-User coccocbot-web OAI-SearchBot Claude


Googlebot Googlebot-Image GoogleOther GoogleOther-Image Google-InspectionTool Yeti Bingbot Daum Slurp Baiduspider ZumBot Storebot-Google Ads-Naver Twitterbot *


* Amazonbot Anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web Cohere-ai DataForSeoBot FacebookBot Google-Extended GPTBot ImagesiftBot Magpie-crawler meta-externalagent NewsNow Omgili Omgilibot Peer39_crawler Peer39_crawler/1.0 PerplexityBot YouBot


*
Disallow: /player/float.php
Disallow: /player/html.php
Disallow: /redirect_cs.php
Disallow: /*?action=*$
Disallow: /*mode=async*$
Disallow: /*function=get_block*$
Disallow: /404.php
Disallow: /logout/
Disallow: /email/
Disallow: /payments/
Disallow: /login/


AhrefsBot
AddSearchBot
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Andibot
anthropic-ai
Applebot
Applebot-Extended
Awario
bedrockbot
bigsur.ai
Brightbot 1.0
Bytespider
CCBot
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
LinerBot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YaK
YandexAdditional
YandexAdditionalBot
YouBot
*


SemrushBot-SA
*
ChatGPT-User
OpenAI
ChatGPT
FacebookBot
Meta AI


* Googlebot-News Mediapartners-Google 008 Baiduspider


ScreamingFrogSEOSpider
Mobian
proximic
Outbrain
Yahoo Ad monitoring
SemrushBot
facebookexternalhit
ias_crawler
identrics
MeltwaterNews
links.streem.com.au
LinkedInBot
redditbot
Mediapartners-Google
Google-InspectionTool
Adsbot-Google
Storebot-Google
AdsBot-Google-Mobile
AdsBot-Google-Mobile-Apps
APIs-Google
ChatGPT-User
GPTBot
OAI-SearchBot
Google-Ads-Creatives-Assistant
Google-adwords
Google-amphtml
Google-cloudvertexbot
Google-play-newsstand
Google-read-aloud
Google-Safety
Google-site-verification
Google-Structured-Data-Testing-Tool
Googleimageproxy
Applebot
AppleNewsBot
adidxbot
admantx
admantx-usbatch/3.1
AmazonAdBot
Chrome-Lighthouse
Verity
Leikibot
parselybot
peer39_crawler/1.0
pingdom
Pinterestbot
PubMatic Crawler Bot
Slack-ImgProxy
Slackbot
Slackbot-LinkExpanding
thetradedesk
Uptimebot
yahoo-linkpreview
Googlebot
Googlebot-Image
Googlebot-News
Googlebot-Video
GoogleOther
GoogleOther-Image
GoogleOther-Video
Bingbot
bingpreview
MicrosoftPreview
DuckDuckBot
Slurp
yahoo
Google-Extended
*


Yandex SemrushBot SemrushBot-SA PetalBot Amazonbot *


* MJ12bot proximic PetalBot Mediapartners-Google IAB-Tech-Lab Verity/1.1


* GPTBot ClaudeBot anthropic-ai Google-Extended Applebot-Extended PerplexityBot Bytespider CCBot Copilot


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


* facebook Twitterbot Slackbot LinkedInBot Whatsapp TelegramBot redditbot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* ia_archiver archive.org_bot heritrix My-heritrix-crawler Timpibot omgili Bytespider DuckAssistBot Applebot-Extended Google-Extended Google-CloudVertexBot GoogleOther GPTBot CCBot Amazonbot FacebookBot cohere-ai cohere-training-data-crawler Omgilibot ClaudeBot Diffbot PanguBot MistralAI-User webz.io DeepSeekBot AmazonAdBot Amzn-SearchBot Amzn-User Claude-SearchBot Claude-User PetalBot Meta-ExternalFetcher Meta-WebIndexer OAI-SearchBot ChatGPT-User PerplexityBot Perplexity-User


Yandex SemrushBot bingbot MJ12bot meta-externalfetcher


* AdsBot-Google Googlebot-Image APIs-Google AdsBot-Google-Mobile


AmazonAdBot Google-InspectionTool ChatGPT-User GPTBot OAI-SearchBot *


* Claude-Web anthropic-ai cohere-ai PerplexityBot seekr Meltwater Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


Sogou web spider
Googlebot
bingbot
MJ12bot
Baiduspider


Baiduspider Baiduspider-image Baiduspider-video Baiduspider-news Baiduspider-favo Baiduspider-cpro Baiduspider-ads Sogou 360Spider YisouSpider YodaoBot Bytespider Ezooms BLEXBot EtaoSpider HuaweiBot QihooSpider MJ12bot Sosospider JikeSpider


* Baiduspider BaiduMobaider BaiduImagespider


*
Mediapartners-Google
Microsoft URL Control
grapeshot
GPTBot
anthropic-ai
CCBot
ChatGPT-User
FacebookBot
Google-Extended
ClaudeBot
Omgilibot
Omgili
Diffbot
Bytespider
ImagesiftBot
cohere-ai


* Amazonbot Bytespider CCBot Cohere-ai Omgili Omgilibot YouBot


* anthropic-ai Claude-Web CCbot FacebookBot Google-Extended GPTBot PiplBot Google-InspectionTool


* AdsBot-Google Baiduspider google-hoteladsverifier


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent Mediapartners-Google


* dotbot dotbot/1.0 Baiduspider MegaIndex.ru AhrefsBot grapeshot Acunetix AwarioRssBot AwarioSmartBot BomboraBot BUbiNG Yandex AmazonAdBot


CCBot anthropic-ai Omgilibot Omgili Diffbot Bytespider ImagesiftBot cohere-ai ClaudeBot Claude-Web Meta-ExternalAgent Meta-ExternalFetcher Applebot Applebot-Extended Amazonbot Meta-WebIndexer GPTBot Google-Extended FacebookBot OAI-SearchBot ChatGPT-User PerplexityBot FirecrawlAgent AndiBot ExaBot PhindBot YouBot *


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


*
SemanticScholarBot
TurnitinBot
MegaIndex.ru
SemrushBot
SemrushBot-SA
Seekport Crawler
ltx71 - (http://ltx71.com/)
AhrefsBot
Riddler
CCBot
ChatGPT-User
Diffbot
GPTBot
Google-Extended
Omgilibot
omgili
FacebookBot
cohere-ai
anthropic-ai
Bytespider
Amazonbot
Applebot
Applebot-Extended
PerplexityBot
YouBot
AdsBot-Google
AwarioRssBot
AwarioSmartBot
ClaudeBot
Claude-Web
DataForSeoBot
FriendlyCrawler
GoogleOther
ImagesiftBot
magpie-crawler
Meltwater
peer39_crawler
PiplBot
Seekr
Meta-ExternalAgent
Timpibot
SummalyBot
facebookexternalhit
ICC-Crawler
img2dataset
OAI-SearchBot
PetalBot
Scrapy
VelenPublicWebCrawler


GPTBot CCBot anthropic-ai Claudebot Bytespider Amazonbot FacebookBot cohere-ai DataForSeoBot Diffbot omgili omgilibot YouBot SemrushBot dotbot MJ12bot Rogerbot Scrapy TurnitinBot magpie-crawler NewsNow news-please AwarioRssBot AwarioSmartBot PiplBot Google-Extended Googlebot Googlebot-News Googlebot-Image Googlebot-Video Googlebot-Mobile PerplexityBot Bingbot Claude-SearchBot Claude-User Petalbot OAI-SearchBot *


AddSearchBot
AhrefsBot
AI2Bot
AihitBot
AllenAI
Amazon-Kendra
Andibot
Anomura
anthropic-ai
Applebot-Extended
archive.org_bot
arquivo-web-crawler
arquivo.pt
Awario
BacklinkCrawler
Baiduspider
Barkrowler
BedrockBot
bigsur.ai
BLEXBot
Bloodhound
Bravebot
Brightbot 1.0
browsertrix
brozzler
Buddybot
BuiltWith
Bytespider
CCBot
ChatGLM-Spider
ChatGPT Agent
Cincraw
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
Cohere-Training-Data-Crawler
Cotoyogi
CrawlSpace
cydralspider
DataForSeoBot
Datenbank Crawler
DeepSeek
DeepSeekBot
Devin
Diffbot
dotbot
downloadexpress
DuckAssistBot
Echobot Bot
EchoboxBot
FactSet_SpyderBot
Fasterfox
FireCrawlAgent
Flamingo_SearchEngine
FriendlyCrawler
gammaSpider
Google-Extended
GoogleAgent-Mariner
GPTBot
ia_archiver
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Jetslide
Kangaroo Bot
Kraken
laion-huggingface-processor
LCC
LinerBot
Linkfluence Yak Bot
LinkWalker
magpie-crawler
MistralAI-User
MistralAI-User/1.0
MJ12bot
msnbot-media/1.1
msnbot-media/2.0b
MyCentralAIScraperBot
netEstate Imprint Crawler
news-please
NewsNow
NovaAct
ObjectsSearch
omgili
omgilibot
Operator
PanguBot
Panscient
panscient.com
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
proximic
QualifiedBot


* PingBot PerplexityBot GPTBot BaiduSpider Meta-ExternalAgent


* GPTBot ChatGPT-User OAI-SearchBot PerplexityBot Amazonbot Bytespider ClaudeBot Claude-Web Meta-ExternalAgent FacebookBot CCBot Anthropic-ai YouBot Google-Extended xAI-Crawler DeepSeek facebookexternalhit Facebot


* Baiduspider AhrefsBot MJ12bot 008 BLEXBot sistrix ia_archiver proximic SiteExplorer


* ClaudeBot GPTBot FacebookBot AhrefsBot CCbot Bytespider cohere-ai Google-Extended Applebot-Extended Scrapy FriendlyCrawler ChatGLM-Spider DataForSeoBot ImagesiftBot TapnativeScrapingBot


Googlebot Googlebot-image AdsBot-Google *


msnbot MJ12bot AhrefsBot Yandex bingbot *


* AI2Bot-Dolma Amazonbot anthropic-ai ApifyBot ApifyWebsiteContentCrawler Applebot-Extended Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler Diffbot FacebookBot FirecrawlAgent Google-Extended GoogleOther GPTBot imageSpider meta-externalagent omgili PanguBot Spider Timpibot Webzio-Extended


Googlebot
AdsBot-Google-Mobile
AdsBot-Google
Mediapartners-Google
AdsBot-Google-Mobile-Apps
Googlebot-Image
Googlebot-Video
Google-Site-Verification
facebookexternalhit
facebookcatalog
Twitterbot
bingbot
Applebot
Yahoo! Slurp
SemrushBot
SiteAuditBot
SemrushBot-BA
SemrushBot-SI
SemrushBot-SWA
SemrushBot-CT
SplitSignalBot
SemrushBot-COUB
seo-audit-check-bot
rogerbot
dotbot
AhrefsBot
AhrefsSiteAudit
OAI-SearchBot
*


* GoogleBot Flipboard FlipboardProxy Yandex


AmazonAdBot * Mediapartners-Google GPTBot CCBots DeepSeek


* Amazonbot anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai DataForSeoBot Diffbot FacebookBot FriendlyCrawler Google-CloudVertexBot Google-Extended GPTBot ImagesiftBot magpie-crawler Meta-ExternalAgent meta-externalagent NewsNow news-please OAI-SearchBot omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot Quora-Bot Scrapy TurnitinBot


* bingbot Baiduspider trovitBot grapeshot


*
Xenu Link Sleuth
Yandex
ICCrawler - iCjobs
Web Downloader/3.0
MJ12bot
AhrefsBot


* bingbot YandexBot SemrushBot dotbot ia_archiver MJ12bot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


* Amazonbot Anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web Cohere-ai DataForSeoBot FacebookBot GPTBot ImagesiftBot Magpie-crawler Omgili Omgilibot Peer39_crawler Peer39_crawler/1.0 PerplexityBot YouBot


*
facebookexternalhit
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Applebot-Extended
Brightbot 1.0
Bytespider
CCBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Diffbot
DuckAssistBot
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
imgproxy
ISSCyberRiskCrawler
Kangaroo Bot
NovaAct
omgili
omgilibot
Operator
PanguBot
Perplexity-User
PerplexityBot
PetalBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
TikTokSpider
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot
BLEXBot
Meltwater
grapeshot


* Storebot-Google Googlebot Googlebot-image


*
uMBot
uMBot-LN
uMBot-IC
Arianna
Arianna News
Arianna Web
Arianna Robot
GPTBot
Google-Extended
anthropic-ai
CCBot
Omgilibot
cohere-ai
ChatGPT-User
OAI-SearchBot
PerplexityBot
Amazonbot
YouBot
Claude-Web
Claudebot
Claude-User
Claude-SearchBot
Diffbot
Bytespider
ImagesiftBot
Meltwater
Seekr


AhrefsBot AwarioRssBot AwarioSmartBot barkrowler BLEXBot CCBot DataForSeoBot dotbot GPTBot ImagesiftBot MJ12bot SemrushBot TurnitinBot * AmazonAdBot


AmazonAdBot GPTBot CCBot ChatGPT-User Google-Extended anthropic-ai cohere-ai omgili omgilibot ICC-Crawler Applebot-Extended ClaudeBot Claude-Web PerplexityBot Perplexity-ai Bytespider Diffbot FacebookBot OAI-SearchBot MJ12bot PiplBot Meta-ExternalAgent Timpibot Webzio Webzio-extended Perplexity-User Claude-SearchBot Claude-User AmazonBot cohere-training-data-crawler magpie-crawler meta-externalfetcher Scrapy DuckAssistBot Google-CloudVertexBot AI2Bot AI2Bot-Dolma img2dataset YouBot SeekrBot OpenindexSpider Gensparkbot SBIntuitionsBot SBIntuitions-SearchBot *


* Google-Extended GPTBot ChatGPT-User CCBot ClaudeBot Bytespider Amazonbot Applebot-Extended ia_archiver Arquivo-web-crawler anthropic-ai OAI-SearchBot PerplexityBot Claude-Web


* TurnitinBot ConveraCrawler QuepasaCreep Jetbot TwengaBot


* AdsBot-Google meta-externalagent facebookexternalhit


proximic
SemrushBot
Twitterbot
SemrushBot-SA
Starkbot
Scrapy
VoluumDSP-content-bot
Seekport Crawler
GPTBot
Google-Extended
*


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* CCBot GPTBot Google-Extended anthropic-ai Omgilibot Omgili FacebookBot Diffbot Bytespider ImagesiftBot cohere-ai PerplexityBot ClaudeBot Claude-Web Meta-ExternalAgent Meta-ExternalFetcher Applebot Applebot-Extended Amazonbot ChatGPT-User OAI-SearchBot Gemini-Deep-Research GoogleOther Google-CloudVertexBot Claude-User Claude-SearchBot Perplexity-User


Amazonbot anthropic-ai Applebot Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User Claude-Web CloudVertexBot cohere-ai DataForSeoBot DeepSeek DeepSeekBot Diffbot FacebookBot FriendlyCrawler Google-CloudVertexBot Google-Extended GPTBot ImagesiftBot magpie-crawler Meta-ExternalAgent meta-externalagent NewsNow news-please OAI-SearchBot omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot Perplexity-User Quora-Bot Scrapy TurnitinBot YandexAdditional YandexAdditionalBot AmazonAdBot *


AI2Bot Ai2Bot-Dolma aiHitBot Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler DuckAssistBot FacebookBot Google-Extended GoogleOther GoogleOther-Image GoogleOther-Video GPTBot img2dataset Meta-ExternalAgent MyCentralAIScraperBot omgili omgilibot Quora-Bot TikTokSpider YouBot AdsBot-Google AdsBot-Google-Mobile AdsBot-Google-Mobile-Apps *


*
Yandex
proximic
ia_archiver
linkdexbot
grapeshot
AhrefsBot
Baiduspider
Clickagy
Clickagy Intelligence Bot v2
Genieo
MJ12bot
ADmantX
crystalsemantics
getintent
Amazon.CloudFront
yoozBot
baidu
WebIndex
Bombora
Sogou
Experibot
HybridBot
Slurp
expo9
CCBot
RealPlayer
weborama
masscan
Cliqzbot
Leikibot


* Google-Extended ChatGPT-User Anthropic-AI Manus


claritybot GPTBot Google-Extended CCBot ChatGPT-User anthropic-ai Omgilibot Omgili FacebookBot Bytespider *


* seekbot Exabot Slurp Cliqzbot Arquivo-web-crawler AI2Bot ChatGLM ChatGLM-spider cohere-training-data-crawler Diffbot OmigiliBot PetalBot YouBot


* Yandex Bingbot peer39_crawler peer39_crawler/1.0 ChatGPT-User GPTBot Applebot-Extended ClaudeBot DataForSeoBot meta-externalagent AmazonAdBot


*
Googlebot-News
MegaIndex.ru/2.0
BLP_bbot/0.1
Slurp
GPTBot
ChatGPT-User
Google-Extended
CCBot
anthropic-ai
Bytespider
ClaudeBot
Claude-Web
cohere-ai
DataForSeoBot
Diffbot
FacebookBot
magpie-crawler
NewsNow
news-please
omgilibot
PerplexityBot
Amazonbot
Applebot-Extended
AwarioRssBot
AwarioSmartBot
FriendlyCrawler
ImagesiftBot
Meta-ExternalAgent
meta-externalagent
OAI-SearchBot
peer39_crawler
peer39_crawler/1.0
Quora-Bot
Scrapy
TurnitinBot
ChatGPT
ICC-Crawler
omgili
Operator
Perplexity-User
PetalBot
TikTokSpider
Timpibot
Webzio-Extended
MistralAI-User
Anchor Browser
Meta-ExternalFetcher


* googlebot yandex googlebot-image googlebot-video yandexvideo


* bingbot YandexBot SemrushBot dotbot ia_archiver MJ12bot


AmazonAdBot Google-InspectionTool ChatGPT-User GPTBot OAI-SearchBot *


ClaudeBot anthropic-ai GPTBot Bingbot SemrushBot SiteAuditBot Amazonbot amazonbot Meta-ExternalAgent meta-externalagent meta-webindexer *


bingbot msnbot Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User Claude-SearchBot Claude-User Claude-Web ClaudeBot cohere-ai cohere-training-data-crawler Diffbot FacebookBot Google-Extended GPTBot ICC-Crawler magpie-crawler Meta-ExternalAgent Meta-ExternalFetcher OAI-SearchBot omgili omgilibot Perplexity Perplexity-ai Perplexity-User PerplexityBot Scrapy Timpibot Webzio-Extended Gensparkbot *


* CazoodleBot MJ12bot dotbot/1.0 Gigabot Semrushbot-SA Semrushbot-SI Caliperbot/1.0 Googlebot Googlebot-image OAI-SearchBot


Crescent Internet ToolPak HTTP OLE Control v.1.0


* AdsBot-Google-Mobile AdsBot-Google PetalBot dotbot AhrefsBot


OAI-SearchBot GPTBot ChatGPT-User Googlebot Google Mediapartners-Google bingbot bing msnbot msn Slurp Yandex Facebot Twitterbot ia_archiver naver NaverBot Yeti *


* GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-Web PerplexityBot Perplexity-User Google-Extended


GPTBot
ChatGPT-User
OAI-SearchBot
ChatGPT Agent
Operator
OpenAI
ClaudeBot
anthropic-ai
Claude-SearchBot
Claude-User
Claude-Web
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
Google-CloudVertexBot
CloudVertexBot
Google-NotebookLM
NotebookLM
GoogleAgent-Mariner
Gemini-Deep-Research
Google-Firebase
Applebot-Extended
Meta-ExternalAgent
meta-externalagent
Meta-ExternalFetcher
meta-externalfetcher
meta-webindexer
FacebookBot
facebookexternalhit
Bytespider
TikTokSpider
Amazonbot
Amzn-SearchBot
Amzn-User
AmazonBuyForMe
amazon-kendra
PerplexityBot
Perplexity-User
DeepSeekBot
MistralAI-User
MistralAI-User/1.0
CCBot
cohere-ai
cohere-training-data-crawler
AzureAI-SearchBot
bedrockbot
Bravebot
DuckAssistBot
PhindBot
YouBot
PetalBot
Diffbot
iAskBot
iaskspider
iaskspider/2.0
kagi-fetcher
LinerBot
LinkupBot
IbouBot
PanguBot
ChatGLM-Spider
Manus-User
NovaAct
Devin
TwinAgent
Scrapy
Crawl4AI
Crawlspace
FirecrawlAgent
img2dataset
LAIONDownloader
laion-huggingface-processor
Cloudflare-AutoRAG
TavilyBot
omgili
omgilibot
AddSearchBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
Andibot
Anomura
atlassian-bot
Awario
bigsur.ai
Brightbot 1.0
BuddyBot
Channel3Bot
Cotoyogi
Datenbank Crawler
Echobot Bot
EchoboxBot
Factset_spyderbot
FriendlyCrawler
ICC-Crawler
ImagesiftBot
imageSpider
ISSCyberRiskCrawler


facebookexternalhit facebookcatalog Twitterbot LinkedInBot WhatsApp Pinterest TelegramBot redditbot *


* Googlebot Bingbot Baiduspider Twitterbot


* GoogleBot Yandex GPTBot ClaudeBot PerplexityBot OAI-SearchBot Googlebot-Extended


* Googlebot Googlebot-Image facebookexternalhit facebookcatalog meta-externalagent meta-externalfetcher Facebot SemrushBot AhrefsBot googlebot-ads mediapartners-google


* Applebot-Extended Bytespider ClaudeBot Claude-Web anthropic-ai archive.org_bot CCBot Arquivo-web-crawler


Linguee Slurp ia_archiver GPTBot ChatGPT-User Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot Claude-Web cohere-ai Diffbot FacebookBot FriendlyCrawler ImagesiftBot img2dataset Meta-ExternalAgent OAI-SearchBot omgili omgilibot PerplexityBot Scrapy Timpibot VelenPublicWebCrawler YouBot *


MJ12bot SemrushBot Amazonbot GoogleOther OAI-SearchBot ClaudeBot GPTBot ImagesiftBot Meta-ExternalAgent meta-webindexer/1.1 *


* CriteoBot/0.1 AmazonAdBot Googlebot ClaudeBot


* Adsbot-Google Mediapartners-Google Slurp


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* Googlebot-Image AdsBot-Google AdsBot-Google-Mobile


* AI2Bot Amazonbot Applebot-Extended anthropic-ai Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai Diffbot DuckAssistBot FacebookBot Google-Extended GPTBot Meta-ExternalAgent Meta-ExternalFetcher OAI-SearchBot PerplexityBot PerplexityUser


* GPTBot Amazonbot Google-Extended CCBot anthropic-ai


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Yandex SindiceBot CCBot wget WebReaper AhrefsBot


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


*
AI2Bot
anthropic-ai
Applebot-Extended
Arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
coccocbot-web
cohere-ai
cohere-training-data-crawler
Crawlspace
Diffbot
DuckAssistBot
FriendlyCrawler
Gensparkbot
Google-CloudVertexBot
Google-Extended
GPTBot
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
magpie-crawler
MoodleBot
Netvibes
NewsNow
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
Scrapy
Seekr
SemrushBot
Timpibot
Turnitin
TurnitinBot
VelenPublicWebCrawler
Webzio-Extended
yacy
yacybot
YouBot
Baiduspider


AiHitBot
BLEXBot
CareerBot
CCBot
dotbot
Google-Extended
GPTBot
grapeshot
iCjobs
linkdexbot
magpie-crawler
MegaIndex
MJ12bot
proximic
QuerySeekerSpider
scrapy
scrapybot
SemrushBot
sentibot
SEOkicks-Robot
TkBot
trendkite-akashic-crawler
Vagabondo
WBSearchBot
AhrefsBot
Baiduspider
YandexBot
Yahoo! Slurp
*
facebookexternalhit


*
Googlebot
Mediapartners-Google
Adsbot-Google
Googlebot-Image
Googlebot-Mobile
Domain Re-Animator Bot
ClaudeBot
Claude-User
Claude-SearchBot
GPTBot
OAI-SearchBot
ChatGPT-User
PerplexityBot
Perplexity-User
Google-Extended


Googlebot Googlebot-Image YandexBox DuckDuckBot Slurp MSNBot MSNBot-Media Bingbot SemrushBot AhrefsBot Twitterbot *


* BLEXBot Mediapartners-Google Googlebot-Mobile GPTBot CCBot


Screaming Frog SEO Spider
facebookexternalhit
*


AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Andibot
anthropic-ai
Applebot
Applebot-Extended
Awario
bedrockbot
Brightbot 1.0
Bytespider
CCBot
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
omgilibot
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
Sidetrade indexer bot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YandexAdditional
YandexAdditionalBot
YouBot
*


* OAI-SearchBot ChatGPT-User GPTBot anthropic-ai ClaudeBot PerplexityBot Google-Extended BingBot


Googlebot BLEXBot dotbot semrushbot ahrefsbot mj12bot barkrowler siteliner linkbot cliqzbot


*
Teleport
TeleportPro
EmailCollector
EmailSiphon
WebBandit
WebZIP
WebReaper
WebStripper
Web Downloader
WebCopier
Offline Explorer Pro
Offline Explorer
HTTrack Website Copier
Offline Commander
Leech
WebSnake
BlackWidow
HTTP Weazel


* Googlebot Bingbot Slurp DuckDuckBot Meta-ExternalAds OAI-SearchBot GPTBot ChatGPT-User ClaudeBot Claude-User PerplexityBot Perplexity-User Google-Extended CCBot Bytespider


Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)


*
magpie-crawler
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Andibot
anthropic-ai
Applebot-Extended
Awario
bedrockbot
Brightbot 1.0
Bytespider
CCBot
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
meta-externalagent
Meta-ExternalAgent
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
omgilibot
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
SummalyBot
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YandexAdditional
YandexAdditionalBot
YouBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


OpenAI-Crawler Googlebot-Bard GPTBot ChatGPT-User Google-Extended CCBot facebookexternalhit *


* schibstedsokbot facebookexternalhit Facebot


*
CazoodleBot
MJ12bot
dotbot/1.0
Gigabot
SEMrushBot
Yandex
Baiduspider
DotBot
MegaIndex.ru
BLEXBot
SEOkicks-Robot
Exabot
AlphaBot
Sogou web spider
Ezooms
LinkpadBot
WebMeUp


* bingbot GoogleOther MJ12bot Pinterestbot YandexBot


AdsBot-Google AdsBot-Google-Mobile AdsBot-Google-Mobile-Apps Mediapartners-Google Googlebot-Image Storebot-Google Sentibot *


* grapeshot AhrefsBot Rogerbot Dotbot MJ12bot SemrushBot SemrushBot-SA PetalBot Bytespider


*
CCBot
GPTBot
anthropic-ai
Google-Extended
Amazonbot
Claude-SearchBot
Meta-ExternalAgent
Bytespider
ClaudeBot
Timpibot
OAI-SearchBot
Claude-User
Applebot-Extended
AI2Bot
Ai2Bot-Dolma
Anomura
bedrockbot
Channel3Bot
ChatGLM-Spider
cohere-ai
cohere-training-data-crawler
Crawl4AI
Crawlspace
DeepSeekBot
Diffbot
FirecrawlAgent
FriendlyCrawler
GoogleOther
GoogleOther-Image
GoogleOther-Video
GrokBot
ICC-Crawler
img2dataset
ImagesiftBot
Kangaroo Bot
KunatoCrawler
laion-huggingface-processor
LAIONDownloader
meta-webindexer
MyCentralAIScraperBot
omgili
omgilibot
PanguBot
SBIntuitionsBot
Scrapy
Spider
TerraCotta
TikTokSpider
VelenPublicWebCrawler
Webzio-Extended
xAI-Grok
YandexAdditional
YandexAdditionalBot
Google-CloudVertexBot
PerplexityBot
Amzn-SearchBot
AzureAI-SearchBot
Bravebot
CloudVertexBot
Grok-DeepSearch
iAskBot
iaskspider
iaskspider/2.0
LinerBot
LinkupBot
PhindBot
TavilyBot
WRTNBot
YouBot
ChatGPT-User
Perplexity-User
MistralAI-User
Meta-ExternalFetcher
AmazonBuyForMe
Amzn-User
ChatGPT Agent
Devin
Gemini-Deep-Research
Google-NotebookLM
GoogleAgent-Mariner
Manus-User
NovaAct
Operator
Claude-Web


Mediapartners-Google* grapeshot YahooSeeker* MSNBOT_Mobile AmazonAdBot *


Crescent Internet ToolPak HTTP OLE Control v.1.0


* ImagesiftBot ClaudeBot Claude-User Claude-SearchBot


yahoo-mmcrawler psbot asterias yahoo-blogs/v3.9 * GPTBot ChatGPT-User ClaudeBot Bytespider


* Twitterbot GPTBot ChatGPT-User CCBot anthropic-ai Amazonbot Applebot-Extended Bytespider ClaudeBot Claude-Web cohere-ai Diffbot Google-Extended Meta-ExternalFetcher PerplexityBot YouBot DeepSeekBot MistralAI-User MistralAI-User/1.0 PetalBot Timpibot YandexAdditional YandexAdditionalBot YandexBot YandexRenderBot Perplexity-User coccocbot-web OAI-SearchBot Claude


*
facebookexternalhit
FacebookBot
GDdash SEO Spider/2.0
Flamingo_SearchEngine
ExaBot
dotbot
genieo
crystalsemantics
grapeshot
SEMrushBot
Scrapy
AhrefsBot
DataForSeoBot
ClaudeBot
Claude-Web
cohere-ai
Diffbot
magpie-crawler
NewsNow
news-please


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


Googlebot Yeti Daumoa ChatGPT-User GPTBot grapeshot *


GPTBot Applebot-Extended anthropic-ai Bytespider CCBot ClaudeBot Diffbot FacebookBot Meta-ExternalAgent omgili Timpibot Webzio-Extended Mediapartners-Google Adsbot-Google AmazonAdbot Amazonbot Amzn-SearchBot Amzn-User amazon-kendra amazon-QBusiness bedrockbot aws-quick-on-behalf-of- AmazonProductDiscoverybot AmazonBuyForMe AmazonSellerInitiatedListing NovaAct DDM-DCipher/1.0.7 *


* anthropic-ai ChatGPT-User ClaudeBot cohere-ai GPTBot OAI-SearchBot PerplexityBot CCbot Googlebot Googlebot-Image Googlebot-Video Googlebot-News AdsBot-Google AdsBot-Google-Mobile Google-Extended Bingbot DuckDuckGo Applebot facebot Yandex Baiduspider Sogou Naverbot MojeekBot PetalBot AhrefsBot SemrushBot MJ12bot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent Omgilibot Omgili DeepSeekBot DeepSeek DuckAssistBot OAI-Searchbot ChatGPT-User Perplexity discobot dotbot TurnitinBot magpie-crawler yacybot Timpibot Nutch grapeshot adsbot-google AhrefsBot AhrefsSiteAudit MJ12bot Pinterest


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* meta-externalagent Meta-ExternalFetcher ChatGPT-User OAI-SearchBot Bytespider DuckAssistBot Perplexity-User Timpibot ClaudeBot Amazonbot GPTBot PerplexityBot CCBot FacebookBot Ai2Bot-Dolma Claude-SearchBot Claude-User omgilibot TtoYouBot WebGPTBot


* anthropic-ai cohere-ai omgili omgilibot Applebot-Extended Bytespider CCBot ChatGPT-User ChatGPT DeepSeek DeepSeekBot GPTBot OAI-SearchBot ClaudeBot Claude-Web Diffbot FacebookBot Google-Extended Meta-ExternalAgent Meta-ExternalFetcher PerplexityBot Timpibot Webzio-Extended YouBot


Amazonbot
Applebot
GPTBot
A6-Indexer
AhrefsBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot-Extended
Baiduspider
betaBot
BLEXBot
botify
brands-bot-logo
Brightbot
Bytespider
calculon
CCBot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
collection@infegy.com
Cotoyogi
Crawl4AI
CrunchBot
deepcrawl
DeepSeekBot
Diffbot
DirBuster
DTS
DuckAssistBot
EasouSpider
FacebookBot
FirecrawlAgent
FriendlyCrawler
GlutenCrawler
Google-CloudVertexBot
Google-Extended
GrapeshotCrawler
iaskspider
ICC-Crawler
ImagesiftBot
img2dataset
JackCrawl
Jetslide
linkapedia_bot
Mediatoolkitbot
MegaIndex
meta-externalagent
meta-externalfetcher
Microsoft BITS
Microsoft URL Control
Microsoft-WebDAV-MiniRedir
MistralAI-User
news-please
Novellum
NTENTbot
Nutch
omgili
Omgilibot
OpenLinkProfiler
Perplexity-User
PerplexityBot
PetalBot
Primalbot
python-requests
Re-Animator
Riddler
SafeDNSBot
SBIntuitionsBot
Scrapy
SearchAtlas
SemanticScholarBot
SemrushBot
SentiBot
Sogou
Spider-Pig
sqlmap
StorygizeBot
Test Certificate Info
TikTokSpider
Timpibot
Turnitin
uipbot
vBSEO
VegeBot
VelenPublicWebCrawler
Web Link Validator
WebCruiser


ChatGPT-User PerplexityBot GPTBot anthropic-ai CCBot Claude-Web ClaudeBot cohere-ai YouBot Diffbot OAI-SearchBot meta-externalagent Timpibot Amazonbot Bytespider Perplexity-User *


Googlebot-News
Mediapartners-Google
AdsBot-Google
proximic
ia_archiver
008
AhrefsBot
baiduspider
baiduspider-video
baiduspider-image
bytespider
DataForSeoBot
discoverybot
DotBot
Exabot
ichiro
KunatoCrawler
MJ12bot
moget
NaverBot
psbot
PaperLiBot
PetalBot
PiplBot
ScooperBot
SemrushBot
sogou spider
Sosospider
Yandex
YandexImages
Yeti
YoudaoBot
Spinn3r
anthropic-ai
CCBot
Claude-Web
Google-Extended
GPTBot
PerplexityBot
*


ia_archiver-web.archive.org ia_archiver archive.org_bot * AdsBot-Google Google-PageRenderer OAI-SearchBot GPTBot ChatGPT-User Bytespider PetalBot ISSCyberRiskCrawler img2dataset


Googlebot Bingbot Twitterbot facebookexternalhit *


Yandex * OAI-SearchBot GPTBot ChatGPT-User ClaudeBot PerplexityBot Applebot-Extended Google-Extended CCBot


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


* baiduspider baiduspider+ Yandex GPTBot AmazonAdBot


* 8LEGS AhrefsBot AspiegelBot BLEXBot Barkrowler DotBot MJ12bot MauiBot Nimbostratus-Bot PetalBot SemrushBot SeznamBot Sogou serpstatbot trendiction TextBulkerBot Amazonbot Bytespider CCBot cohere-ai FacebookBot Omgilibot


Mediapartners-Google*
TruliaBot
Twitterbot
*
Amazonbot
Applebot
GPTBot
A6-Indexer
AhrefsBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot-Extended
Baiduspider
betaBot
BLEXBot
botify
brands-bot-logo
Brightbot
Bytespider
calculon
CCBot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
collection@infegy.com
Cotoyogi
Crawl4AI
CrunchBot
deepcrawl
DeepSeekBot
Diffbot
DirBuster
DTS
DuckAssistBot
EasouSpider
FacebookBot
FirecrawlAgent
FriendlyCrawler
GlutenCrawler
Google-CloudVertexBot
Google-Extended
GrapeshotCrawler
iaskspider
ICC-Crawler
ImagesiftBot
img2dataset
JackCrawl
Jetslide
linkapedia_bot
Mediatoolkitbot
MegaIndex
meta-externalagent
meta-externalfetcher
Microsoft BITS
Microsoft URL Control
Microsoft-WebDAV-MiniRedir
MistralAI-User
news-please
Novellum
NTENTbot
Nutch
omgili
Omgilibot
OpenLinkProfiler
Perplexity-User
PerplexityBot
PetalBot
Primalbot
python-requests
Re-Animator
Riddler
SafeDNSBot
SBIntuitionsBot
Scrapy
SearchAtlas
SemanticScholarBot
SemrushBot
SentiBot
Sogou
Spider-Pig
sqlmap
StorygizeBot
Test Certificate Info
TikTokSpider
Timpibot
Turnitin
uipbot
vBSEO


* YandexBot MJ12bot SiteimproveBot-Crawler SiteimproveBot


Googlebot Slurp Bingbot msnbot gsa-crawler *


Mediapartners-Google* archive.org_bot BoardReader GPTBot Google-Extended PerplexityBot CCBot FacebookBot OmgiliBot anthropic-ai cohere-ai *


AhrefsBot AlphaBot Baiduspider Baiduspider-render Buck changedetection Cliqzbot Exabot Flamingo_SearchEngine Grobbot JobboerseBot jobs.de-Robot linkdexbot Mail.RU_Bot Mediatoolkitbot MegaIndex.ru/2.0 Mega-Index MegaIndex.ru MJ12bot MojeekBot Pinterestbot rogerbot SafeDNSBot SemrushBot SemrushBot/2~bl SemrushBot-BA SemrushBot-BM SemrushBot-SA SEOkicks-Robot SeznamBot SMTBot spbot trendictionbot TweetmemeBot vebidoobot wonderbot Wotbox yacybot YaK YandexAntivirus YandexBot/3.0 YandexImages YandexMobileBot Yeti


* AmazonAdBot bytespider mistralai cohere ai2bot youbot omgili diffbot kangaroo img2dataset amazonbot amazon-qbusiness meta-externalagent meta-webindexer


* ia_archiver archive.org_bot heritrix My-heritrix-crawler Timpibot omgili Bytespider DuckAssistBot Applebot-Extended Google-Extended Google-CloudVertexBot GoogleOther GPTBot CCBot Amazonbot FacebookBot cohere-ai cohere-training-data-crawler Omgilibot ClaudeBot Diffbot PanguBot MistralAI-User webz.io DeepSeekBot AmazonAdBot Amzn-SearchBot Amzn-User Claude-SearchBot Claude-User PetalBot Meta-ExternalFetcher Meta-WebIndexer OAI-SearchBot ChatGPT-User PerplexityBot Perplexity-User


* anthropic-ai Claude-Web CCBot FacebookBot Google-Extended GPTBot PiplBot


OAI-SearchBot ChatGPT-User PerplexityBot FirecrawlAgent AndiBot ExaBot PhindBot YouBot GPTBot CCBot Google-Extended Googlebot Bingbot Googlebot-Image Googlebot-Video Googlebot-Mobile *


* anthropic-ai ChatGPT-User ClaudeBot Claude-Web cohere-ai GPTBot OAI-SearchBot quillbot.com PerplexityBot TurnitinBot AmazonAdBot


* SemrushBot DataForSeoBot SemrushBot-SA dotbot AhrefsBot MJ12bot GPTBot


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


* Baiduspider Baiduspider-image Baiduspider-video Baiduspider-news Baiduspider-favo Baiduspider-cpro Baiduspider-ads MSNBot MSNBot-Media AdIdxBot GPTBot OAI-SearchBot Google-Extended meta-externalagent Amazonbot GoogleOther Anthropic-AI Claude-Web ClaudeBot Claude-SearchBot PerplexityBot Cohere cohere-ai Applebot-Extended Google-CloudVertexBot DotBot spbot AhrefsBot CCBot Omgilibot Omgili FacebookBot Bytespider Bytedance Diffbot Youbot FriendlyCrawler img2dataset ChatGPT-User Perplexity-User Claude-User meta-webindexer facebookexternalhit


* Mediapartners-Google bingbot meta-externalagent


Crescent Internet ToolPak HTTP OLE Control v.1.0


*

Disallow: /player/float.php
Disallow: /player/html.php
Disallow: /redirect_cs.php
Disallow: /*?action=*$
Disallow: /*mode=async*$
Disallow: /*function=get_block*$
Disallow: /404.php
Disallow: /logout/
Disallow: /email/
Disallow: /payments/
Disallow: /login/


* FacebookExternalHit Twitterbot Pinterestbot LinkedInBot TikTokSpider Storebot-Google Google-InspectionTool AmazonAdBot Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended Amazonbot meta-webindexer meta-externalagent meta-externalfetcher ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User cohere-training-data-crawler cohere-ai CCBot PanguBot PetalBot Bytespider Diffbot DuckAssistBot MistralAI-User Timpibot Webzio Webzio-Extended archive.org_bot ia_archiver ia_archiver-web.archive.org heritrix


* AdsBot-Google-Mobile AdsBot-Google PetalBot dotbot AhrefsBot


AmazonAdBot * Arquivo-web-crawler AI2Bot ChatGLM ChatGLM-spider cohere-training-data-crawler Diffbot OmigiliBot PetalBot YouBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


OpenAI-Crawler Googlebot-Bard GPTBot ChatGPT-User Google-Extended CCBot *


bingbot Googlebot YandexBot SemanticScholarBot Applebot Baiduspider DuckDuckBot PetalBot facebookexternalhit Twitterbot *


Googlebot Bingbot Twitterbot facebookexternalhit *


* GPTBot ClaudeBot Google-Extended Anthropic-AI Bytespider CCBot Applebot-Extended Meta-ExternalAgent OAI-SearchBot PerplexityBot


AI2Bot
ChatGLM
ChatGLM-Spider
cohere-training-data-crawler
Diffbot
OmigiliBot
PetalBot
YouBot
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
GPTBot
Meta-ExternalAgent
OAI-SearchBot
Perplexity-User
PerplexityBot
larbin
strucr
psbot
Python-urllib
lwp-trivial/1.34
lwp-trivial
lwp-request
MagpieRSS
Custo
ichiro
Sensis Web Crawler
IRLbot
Exabot
CFNetwork
TencentTraveler
Yandex
AhrefsBot
*


*
MJ12bot
DataForSeoBot
BLEXBot
AhrefsBot
barkrowler
Screaming Frog SEO Spider
ZoomBot
magpie-crawler
DotBot
rogerbot
SemrushBot
SiteAuditBot
SemrushBot-BA
SemrushBot-SI
SemrushBot-SWA
SplitSignalBot
SemrushBot-OCOB
JamesBOT
Oncrawl
AwarioRssBot
AwarioSmartBot
AwarioBot
serpstatbot
netEstate NE Crawler
panscient.com
AcademicBotRTU
TurnitinBot
CheckMarkNetwork
SEOkicks
Timpibot
Seobility
BitSightBot
linkfluence
adbeat_bot
BrandVerity
peer39_crawler
PiplBot
Scrapy
BUbiNG
undici
YisouSpider
TerraCotta
HaloBot
SERankingBacklinksBot
AffsignalCrawler
RankzonBot


* AdsBot-Google AdsBot-Google-Mobile-Apps Mediapartners-Google


* AhrefsBot SemrushBot Yandex YandexBot GPTBot


Twitterbot
libwww-perl
*
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Diffbot
DuckAssistBot
FriendlyCrawler
GPTBot
iaskspider/2.0
ICC-Crawler
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
Meta-ExternalAgent
Meta-ExternalFetcher
OAI-SearchBot
omgili
omgilibot
PanguBot
PetalBot
Scrapy
SemrushBot
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot


* Bytespider YandexBot Spinn3r Yandex Baidu Mail.RU_Bot CCBot omgilibot omgili Sogou Exabot Coccocbot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


Yandex * BLEXBot MJ12bot SemrushBot PetalBot CriteoBot proximic grapeshot ia_archiver


*
anthropic-ai
Google-Extended
GPTBot
Applebot-Extended
cohere-ai
Diffbot
01h4x.com
360Spider
404checker
404enemy
80legs
ADmantX
AIBOT
ALittle Client
ASPSeek
Abonti
Aboundex
Aboundexbot
Acunetix
AdsTxtCrawlerTP
AfD-Verbotsverfahren
AhrefsBot
AiHitBot
Aipbot
Alexibot
AllSubmitter
Alligator
AlphaBot
Anarchie
Anarchy
Anarchy99
Ankit
Anthill
Apexoo
Aspiegel
Asterias
Atomseobot
Attach
AwarioRssBot
AwarioSmartBot
BBBike
BDCbot
BDFetch
BLEXBot
BackDoorBot
BackStreet
BackWeb
Backlink-Ceck
BacklinkCrawler
Badass
Bandit
Barkrowler
BatchFTP
Battleztar Bazinga
BetaBot
Bigfoot
Bitacle
Black Hole
BlackWidow
Blackboard
Blow
BlowFish
Boardreader
Bolt
BotALot
Brandprotect
Brandwatch
Buck
Buddy
BuiltBotTough
BuiltWith
Bullseye
BunnySlippers
BuzzSumo
Bytespider
CATExplorador
CCBot
CODE87
CSHttp
Calculon
CazoodleBot
Cegbfeieh
CensysInspect
CheTeam
CheeseBot
CherryPicker
ChinaClaw
Chlooe
Citoid
Claritybot
Cliqzbot
Cloud mapping
Cocolyzebot
Cogentbot
Collector
Copier
CopyRightCheck
Copyscape
Cosmos


* Googlebot-Mobile AdsBot-Google-Mobile bingbot-mobile Mediapartners-Google APIs-Google AdsBot-Google Googlebot-Image


* GPTBot OAI-SearchBot ClaudeBot PerplexityBot Google-Extended


Crescent Internet ToolPak HTTP OLE Control v.1.0


* Googlebot Mediapartners-Google Googlebot-Image Claude-Web anthropic-ai cohere-ai PerplexityBot seekr Meltwater Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* carbon-umbrella-bot/1.0 CCBot GPTBot Applebot-Extended Google-Extended Meta-ExternalAgent ClaudeBot anthropic-ai Claude-Web cohere-ai omgili Timpibot Webzio-Extended Bytespider ICC-Crawler


adidxbot AdsBot-Google AhrefsBot AliyunSecBot AmazonAdBot baiduspider baiduspider-mobile bingbot BLEXBot Googlebot Googlebot-Mobile GumGum GumGum-Bot Mediapartners-Google MJ12bot msnbot PetalBot slurp weborama weborama-fetcher Yandex Bytespider *


facebookexternalhit GPTBot Google-Extended PerplexityBot *


AhrefsBot proximic TTD-Content Mediapartners-Google *


* Googlebot Googlebot-Image Mediapartners-Google Adsbot-Google AdsBot-Google-Mobile-Apps Googlebot-News Googlebot-Video AdsBot-Google-Mobile


* dotbot rogerbot GPTBot Google-Extended


Bytespider AhrefsBot SemrushBot GPTBot ClaudeBot CCBot Google-Extended PerplexityBot Amazonbot Applebot-Extended PetalBot YandexBot MJ12bot DotBot *


Mediapartners-Google Adsbot-Google Googlebot-Image * msnbot CCBot Baiduspider Sogou BLEXBot


* GPTBot Google-Extended ClaudeBot PerplexityBot Applebot-Extended


AmazonAdBot
admantx
outbrain
criteo
Scope3
thetradedesk
ozone
ApocalXExplorerBot
asterias
BackDoorBot/1.0
Black Hole
BlowFish/1.0
BotALot
BuiltBotTough
Bullseye/1.0
BunnySlippers
Cegbfeieh
CheeseBot
CherryPicker
CherryPickerElite/1.0
CherryPickerSE/1.0
cosmos
Crescent
DIIbot
DISCo Pump 3.1
Download Ninja
downloadexpress
eCatch
EmailCollector
EmailSiphon
EmailWolf
EroCrawler
Explore
ExtractorPro
Flamingo_SearchEngine
Foobot
gammaSpider
Harvest/1.5
hloader
humanlinks
indexer
InfoNaviRobot
JennyBot
Jetbot
k2spider
Kenjin Spider
LexiBot
LinkextractorPro
Mata Hari
NetAnts
NPBot
ObjectsSearch
Offline Explorer
Pimptrain
ProPowerBot/2.14
ProWebWalker
QuepasaCreep
QueryN Metasearch
RepoMonkey
RMA
Roverbot
SightupBot
SiteSnagger
SiteSucker
SpankBot
spanner
suggybot
SuperBot
Teleport
TeleportPro
Telesoft
vsw
wapspider
WebBandit
WebCopier
AgentCore-Browser
Bloodhound
Cotoyogi
CotoyogiBot
Datenbank Crawler
DownloadNinja
ICC-Crawler
toCrawl/UrlDispatcher
WARDBot
WebSauger
website extractor
Website Quester
Webster Pro
WebStripper/2.02
WebZinger
WebZip
WebZip/4.0
WWW-Collector-E
YRSPider
ZanistaBot
Zealbot
Zeus 32297 Webster Pro V2.9 Win32
Zookabot
ZyBORG
Sogou web spider


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* MJ12bot Twitterbot PetalBot Riddler Baiduspider AhrefsBot Amazonbot ClaudeBot


* GPTBot ClaudeBot PerplexityBot Google-Extended


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper


stress-agent Googlebot GPTBot CCBot ClaudeBot Ultraseek ChatGPT-User Doximity *


eBay Relevance Ad Crawler powered by contentDetection (www.mindup.de)

Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)

Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


* 008 proximic ia_archiver CCBot voltron TurnitinBot Paracrawl GPTBot Bytespider


* Baiduspider Googlebot bingbot msnbot Charlotte GPTBot ChatGPT-User Google-Extended CCBot PerplexityBot anthropic-ai Claude-Web ClaudeBot ChatGLM-Spider Bytespider


Googlebot AdsBot-Google AdsBot-Google-mobile Googlebot-image OAI-SearchBot *


* Mediapartners-Google trovitBot pricebot magpie-crawler SemrushBot trendictionbot BLEXBot Riddler


*
AI2Bot
Ai2Bot-Dolma
anthropic-ai
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
ClaudeBot
Claude-Web
cohere-ai
cohere-training-data-crawler
Diffbot
FacebookBot
Google-Extended
GPTBot
Kangaroo Bot
Meta-ExternalAgent
meta-externalagent
Meta-ExternalFetcher
meta-externalfetcher
omgili
omgilibot
Timpibot
Webzio-Extended
magpie-crawler
Scrapy


Googlebot Google-InspectionTool Bingbot BingPreview *


* GPTBot ClaudeBot Google-Extended Bytespider Amazonbot PerplexityBot CCBot anthropic-ai cohere-ai


AmazonAdBot * AI2Bot ChatGLM ChatGLM-spider cohere-training-data-crawler Diffbot OmigiliBot PetalBot YouBot


AhrefsBot SemrushBot anthropic-ai GPTBot PetalBot AmazonAdBot *


OAI-SearchBot
ChatGPT-User
ChatGPT-User/2.0
GPTBot
ClaudeBot
anthropic-ai
Claude-Web
Gemini-Deep-Research
Google-Extended
Applebot-Extended
Meta-ExternalAgent
FacebookBot
PerplexityBot
Bingbot
cohere-ai
CCBot
YouBot
BraveBot
Googlebot
Googlebot-Image
Slurp
DuckDuckBot
Yandex
Applebot
Twitterbot
LinkedInBot
Pinterest
Slackbot
TelegramBot
WhatsApp
Discordbot
*
HTTrack
WebCopier
WebZIP
WebStripper
SiteSnagger
Teleport
TeleportPro
WebCapture
Offline Explorer
Wget
curl
DotBot
MJ12bot
BLEXBot
DataForSeoBot
SeznamBot
PetalBot
Bytespider
serpstatbot
SEOkicks
SEOkicks-Robot
sistrix
JEOD
libwww-perl
Python-urllib
Scrapy
Go-http-client
Java
Jakarta
Mechanize
Nutch
MegaIndex
ltx71
BLP_bbot
CompSpyBot
ChinaClaw
Harvest
EmailCollector
EmailSiphon
EmailWolf
ExtractorPro
Xenu
Zeus
UptimeBot
UptimeRobot
StatusCake
Pingdom
Site24x7
Baiduspider
Sogou
YisouSpider
360Spider
Exabot
GrapeshotCrawler
Proximic
SafeDNSBot
Cliqzbot
magpie-crawler
Ezooms
ia_archiver
archive.org_bot
Wayback
Nikto
sqlmap
Nessus
OpenVAS
Acunetix
Nmap


* GPTBot ChatGPT-User CCBot anthropic-ai ClaudeBot Claude-Web FacebookBot PerplexityBot Cohere-ai Seekr Amazonbot Bytespider BLEXBot ImagesiftBot OAI-SearchBot Applebot YandexBot GPTbot meta-externalagent


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Yandex AdsBot-Google facebookexternalhit Twitterbot


Googlebot
Googlebot-News
Googlebot-Image
Googlebot-Video
GoogleOther
*
BaiduSpider
CompSpyBot
Curious George
CybEye.com
DoCoMo
ExB Language Crawler
Ezooms
Flamingo_SearchEngine
Genieo
Genio
LWNutch
LexxeBot
OpenWebIndex
RediffNewsBot
SEOENGWorldBot
Scanmine
ShopWiki
ShowyouBot
Sosospider
WocBot
YoudaoBot
daumoa
gsa-crawler
libcrawl
linkdex
magpie-crawler
repparser
rogerbot
sindice-site-manager
sogou spider
sogou
woriobot
yacybot
yolinkBot
Nutch
Qwantify
trendkite-akashic-crawler
JamesBOT
BUbiNG
SearchmetricsBot
arquivo-web-crawler
www.analyticsseo.com/crawler
HubSpot
dotbot
Cliqzbot
spbot
NTENTbot
SocialRankIOBot
Mediatoolkitbot
Companybook Crawler
Domain Re-Animator Bot
Mail.RU_Bot
heritrix


Googlebot-News Googlebot * Googlebot-Image AmazonAdBot


Crescent Internet ToolPak HTTP OLE Control v.1.0


* AmazonAdBot Hootsuite HootsuiteBot OgScrper OgScraper Hootsuite-Authoring