* GPTBot ChatGPT-User Google-Extended CCBot anthropic-ai Claude-Web ClaudeBot Applebot-Extended Amazonbot Bytespider cohere-ai FacebookBot MistralAI-User MistralAI-Index Meta-ExternalAgent Meta-ExternalFetcher OAI-SearchBot PerplexityBot


Crescent Internet ToolPak HTTP OLE Control v.1.0


*
msnbot
bingbot
Arianna
Arianna News
Arianna Web
Arianna Robot
Webnews Arianna
magpie-crawler


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


*
Yandex
bingbot
SemrushBot
ia_archiver
ia_archiver-web.archive.org
MJ12bot
AhrefsBot
Seekport Crawler


GPTBot CCBot anthropic-ai Claude-Web Google-Extended PerplexityBot Bytespider Amazonbot Omgilibot Meta-ExternalAgent Diffbot ImagesiftBot cohere-ai MJ12bot dotbot coccocbot-web BLEXBot HTTrack Slurp Baiduspider Yandex magpie-crawler SemrushBot SemrushBot-SA AhrefsBot *


PerplexityBot ClaudeBot GPTBot Google-Extended *


* AmazonAdBot anthropic-ai ChatGPT-User ClaudeBot Claude-Web cohere-ai GPTBot PerplexityBot Bytespider


* Twitterbot Amazonbot Anthropic-ai AwarioRssBot AwarioSmartBot Bytespider CCBot Cohere-ai DataForSeoBot ImagesiftBot Magpie-crawler Omgili Omgilibot Peer39_crawler Peer39_crawler/1.0 YouBot


Googlebot-Mobile Y!J-SRD/1.0 Twitterbot Slurp


Baiduspider
Baiduspider-image
Googlebot
MSNBot
bingbot
Sogou web spider
Sogou inst spider
Sogou spider2
Sogou blog
Sogou News Spider
Sogou Orion spider
360Spider
YisouSpider
YoudaoBot
ChinasoSpider
Sosospider
EasouSpider
JikeSpider
*
GPTBot
ChatGPT-User
Claude-Web
Anthropic-AI
PerplexityBot
Google-Extended
Applebot
Bytespider
QwenBot
TongyiBot
DeepSeekBot
DeepSeek-User
Kimi
MoonshotBot
ChatGLM-Spider
Hunyuan
TencentBot
MiniMaxBot
OAI-SearchBot
Doubao
AliyunBot
PanguBot
iFlytekBot
YiBot
SenseBot
Amazonbot
YouBot
Cohere-AI
Meta-ExternalAgent
CCBot


Baiduspider Yandex * Siteimprove Siteimprovebot Siteimprovebot-crawler


* Amazonbot Applebot Applebot-Extended Bytespider CCBot Claude-User ClaudeBot Claude-SearchBot anthropic-ai Google-Extended GPTBot ChatGPT-User cohere-ai meta-externalagent PerplexityBot OAI-SearchBot


* Googlebot bingbot SemrushBot dotbot AhrefsBot YandexBot MJ12bot BlexBot Baiduspider Sogou PetalBot CCBot MauiBot serpstatbot SEOkicks-Robot


* turnitinbot NPBot ia_archiver webzip OmniExplorer_Bot BoardReader GPTBot


*
AI2Bot
Ai2Bot-Dolma
AliyunSecBot
Amazonbot
AmazonBot
anthropic-ai
Applebot-Extended
AudigentAdBot
AwarioRssBot
AwarioSmartBot
BLEXBot
Bytespider
CCBot
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
DataForSeoBot
DeepSeek
DeepSeekBot
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
GPTBot
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Jetslide
Kangaroo Bot
magpie-crawler
Meta-ExternalAgent
meta-externalagent
Meta-ExternalFetcher
meta-externalfetcher
Meta-WebIndexer
meta-webindexer/1.1
MistralAI-User
MyCentralAIScraperBot
news-please
NewsNow
OAI-SearchBot
omgili
omgilibot
PanguBot
peer39_crawler
peer39_crawler/1.0
Perplexity-User
PerplexityBot
Poseidon Research Crawler
quillbot.com
Quora-Bot
Qwam content intelligence
Scrapy
SeekrBot
SeznamHomepageCrawler
Sidetrade indexer bot
TaraGroup Intelligent Bot
TeleportPro
Timpibot
TurnitinBot
VelenPublicWebCrawler
ViennaTinyBot
Webzio-Extended
YouBot


* Yandex Googlebot-image Googlebot Googlebot-Mobile AdsBot-Google AdsBot-Google-Mobile GPTBot Bingbot ia_archiver YandexAdditionalBot


SemrushBot AhrefsBot MJ12bot GPTBot Google-Extended CCBot anthropic-ai ClaudeBot facebookexternalhit *


Crescent Internet ToolPak HTTP OLE Control v.1.0


AhrefsBot AhrefsSiteAudit DataForSeoBot Eyeotabot Mediatoolkitbot msnbot SemrushBot SeznamBot Yandex Amazonbot anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot CCBot ChatGPT-User ClaudeBot Claude-User Claude-Web Claude-SearchBot DeepSeekBot DeepSeek Diffbot DuckAssistBot Google-Extended GPTBot meta-externalagent meta-externalfetcher OAI-SearchBot PerplexityBot Perplexity-User Timpibot YouBot Mediapartners-Google *


* Twitterbot AhrefsBot SemrushBot DotBot MJ12bot serpstatbot SEOkicks Barkrowler BLEXBot SEBot-WA


BingBot * GPTBot ChatGPT-User CCBot PerplexityBot Google-Extended


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


*
IstellaBot
Curious George
ZoomBot
TurnitinBot
Scanbot
oodlebot
rtbtr-adstxt-bot
linkdexbot
Mail.RU_Bot
musobot
YaK
TweetmemeBot
James BOT
trendictionbot
Mediatoolkitbot
SurdotlyBot


* GPTBot ClaudeBot Google-Extended CCBot anthropic-ai


*
AdsBot-Google
Mediapartners-Google
ClaudeBot
CrystalSemanticsBot
DOC
Download Ninja
Fetch
grub-client
HTTrack
k2spider
larbin
libwww
linko
Microsoft.URL.Control
MSIECrawler
NPBot
Offline Explorer
sitecheck.internetseer.com
SiteSnagger
Teleport
TeleportPro
UbiCrawler
WebCopier
WebReaper
WebStripper
WebZIP
Xenu
Zao
Zealbot
ZyBORG


* Mediapartners-Google Googlebot-News grapeshot AmazonAdBot ClaudeBot anthropic-ai GPTBot ChatGPT-User Google-Extended PerplexityBot CCBot


*
Mediapartners-Google
Googlebot-News
CCBot
Meltwater
trendictionbot
trendictionbot0.5.0
trendiction search
um-IC
um-FC
Panalis
LexisNexis
PetalBot


* Googlebot Googlebot-Image Mediapartners-Google Adsbot-Google


* CCBot GPTBot ClaudeBot meta-externalagent OAI-SearchBot PerplexityBot Claude-SearchBot adidxbot bingbot Yandex


* Yandex AdsBot-Google facebookexternalhit Twitterbot


AmazonAdBot Twitterbot * AdsBot-Google-Mobile AdsBot-Google Mediapartners-Google


*
YandexVideoParser
008
BLEXBot
AhrefsBot
Vagabondo
SEOkicks-Robot
ia_archiver
archive.org_bot
special_archiver
MJ12bot
heritrix
netEstate NE Crawler
sistrix
WBSearchBot
QuerySeekerSpider
proximic
SiteExplorer
SemrushBot
SemrushBot-SA
Mail.Ru
Baiduspider
MSNBot
Bingbot
Slurp
Yandex


Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User Claude-Web cohere-ai cohere-training-data-crawler Diffbot FacebookBot Google-Extended GPTBot ICC-Crawler magpie-crawler meta-externalagent meta-externalfetcher OAI-SearchBot omgili omgilibot PerplexityBot Perplexity-User Scrapy Timpibot Webzio-Extended Gensparkbot


*
008
BLEXBot
AhrefsBot
Vagabondo
SEOkicks-Robot
ia_archiver
archive.org_bot
special_archiver
MJ12bot
heritrix
netEstate NE Crawler
sistrix
WBSearchBot
QuerySeekerSpider
proximic
SiteExplorer
SemrushBot
SemrushBot-SA
DotBot
Mail.Ru
Baiduspider
MSNBot
Bingbot
Slurp
Yandex


* Nuclei WikiDo Riddler PetalBot Zoominfobot Go-http-client Node/simplecrawler CazoodleBot dotbot/1.0 Gigabot Barkrowler BLEXBot magpie-crawler


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


* GPTBot ChatGPT-User OAI-SearchBot PerplexityBot ClaudeBot anthropic-ai Google-Extended Applebot Amazonbot Bytespider cohere-ai


* Amazonbot Bytespider CCBot ClaudeBot GPTBot meta-externalagent anthropic-ai YouBot cohere-ai Diffbot ImagesiftBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* FacebookExternalHit Twitterbot Pinterestbot LinkedInBot TikTokSpider Storebot-Google Google-InspectionTool AmazonAdBot Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended Amazonbot meta-webindexer meta-externalagent meta-externalfetcher ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User cohere-training-data-crawler cohere-ai CCBot PanguBot PetalBot Bytespider Diffbot DuckAssistBot MistralAI-User Timpibot Webzio Webzio-Extended archive.org_bot ia_archiver ia_archiver-web.archive.org heritrix


GPTBot
CCBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
claude-web
anthropic-ai
Googlebot
Google-Extended
PerplexityBot
Applebot
Applebot-Extended
DeepSeek
WangluTechBot
Baiduspider
360Spider
Sogou Spider
YisouSpider
Bytespider
YoudaoBot
ToutiaoSpider
*


Rogerbot Exabot MJ12bot Dotbot Gigabot SemrushBot SemrushBot-SA Semrushbot-SI *


* Mediapartners-Google AdsBot-Google AdsBot-Google-Mobile


claritybot GPTBot Google-Extended CCBot ChatGPT-User anthropic-ai Omgilibot Omgili FacebookBot Bytespider *


* Bytespider GPTBot ClaudeBot Amazonbot OAI-SearchBot AliyunSecBot Applebot Meta-ExternalAgent ia_archiver


* GPTBot ChatGPT-User Google-Extended ClaudeBot anthropic-ai PerplexityBot Amazonbot Bytespider cohere-ai


* duggmirror grapeshot SemrushBot AhrefsBot GPTBot


Mozilla/4.0 (compatible; BullsEye; Windows 95)

Crescent Internet ToolPak HTTP OLE Control v.1.0


* Googlebot-Image Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot Google-Extended GPTBot meta-externalagent


*
Baiduspider
PetalBot
yisouspider
008
ChatGPT-User
ChatGPT-User/2.0
PerplexityBot
Perplexity-User
DuckAssistBot
Applebot
bingbot
Googlebot
ClaudeBot
claude-web
Claude-SearchBot
YouBot
Applebot-Extended
MistralAI-User
meta-webindexer
GoogleAgent-Mariner
DotBot
archive.org_bot
Pinterestbot
TelegramBot
kakaotalk-scrap
getstream.io/opengraph-bot
RamblerMail
Brave
OAI-SearchBot
GPTBot
anthropic-ai
Google-Extended
cohere-ai
CCBot
AI2Bot
Amazonbot
Amazonbot-Video
Bytespider
meta-externalagent
Diffbot
omgili
TimpiBot
SeznamBot
Exabot
YandexBot
Sogou
360Spider
YisouSpider
Baiduspider-Render/2.0
MJ12bot
DataForSeoBot
AliyunSecBot
ArchiveTeam
ArchiveTeam ArchiveBot
AwarioBot
ZoominfoBot
ali-implementer
Blueno
BIGO-baiguoyuan
WorksOgCrawler
OnPageBot
DoCoMo
SAMSUNG-SGH-E250
TA-Googlebot


Bytespider DataForSeoBot sitebot Riddlerbot Ocelli linguatools www.integromedb.org/Crawler 008 Exabot psbot turnitinbot NPBot ia_archiver webzip OmniExplorer_Bot BoardReader GPTBot Googlebot Googlebot-Image Googlebot-Video Googlebot-News Mediapartners-Google AdsBot-Google Google-Display-Ads-Bot *


* RyteBot Mediapartners-Google AmazonAdBot DoubleVerifyBot DVBot Leikibot


*
Mediapartners-Google*
Mediapartners-Google
sistrix
SISTRIX Crawler
SISTRIX
SEOkicks-Robot
jobs.de-Robot
AhrefsBot
UnisterBot
SearchmetricsBot
MJ12bot
SurveyBot
SEOdiver
spbot
wotbox
dotbot
meanpathbot
BacklinkCrawler
magpie-crawler
oBot
fr-crawler
BLEXBot
MegaIndex.ru
megaindex.com
CloudServerMarketSpider
trendictionbot
Exabot
careerbot
Lipperhey-Kaus-Australis
seoscanners.net
MetaJobBot
Spiderbot
LinkStats
JobboerseBot
ICCrawler
Plista
Domain Re-Animator Bot
turnitinbot
coccoc
um-IC
mindUpBot
sg-Orbiter
CCBot
Qwantify
YaK
plukkie
SafeDNSBot
360Spider
HaosouSpider
rogerbot
OpenHoseBot
Screaming Frog SEO Spider
ThumbSniper
R6_CommentReader
ImplisenseBot
Cliqzbot
aiHitBot
WBSearchBot


AI2Bot Ai2Bot-Dolma aiHitBot Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler DuckAssistBot FacebookBot Google-Extended GoogleOther GoogleOther-Image GoogleOther-Video GPTBot img2dataset Meta-ExternalAgent MyCentralAIScraperBot omgili omgilibot Quora-Bot TikTokSpider YouBot AdsBot-Google AdsBot-Google-Mobile AdsBot-Google-Mobile-Apps *


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
googlebot
googlebot-image
GoogleOther
Googlebot-Video
Google-InspectionTool
AdsBot-Google
AdsBot-Google-Mobile
AdsBot-Google-Mobile-Apps
Storebot-Google
google-read-aloud
mediapartners-google
bingbot
msnbot
bingpreview
slurp
yahoo
yandex
YandexImages
baiduspider
Pinterestbot
Yeti
MojeekBot
360Spider
google-cloudvertexbot
duckduckbot
petalbot
Applebot
AppleNewsBot
flipboard
qwantbot
SeznamBot
yisouspider
proximic
admantx
thetradedesk
outbrain
ias_crawler
AmazonAdBot
pubmatic
smartologybot
parselybot
Screaming Frog SEO Spider
DeepCrawl
AhrefsBot
SemrushBot
SimilarWebBot
SISTRIX
OnCrawl
botify
Chrome-Lighthouse
ChatGPT-User
GPTBot
OAI-SearchBot
facebookexternalhit
facebot
twitterbot
linkedinbot
telegrambot
snapchat
newrelicpinger
siteimprovebot
uptimerobot
pingdom.com_bot_version_1.4
sentry
schema-validator
googleimageproxy
Iframely
Vocabtracker
EpvzCrawl6194680250
Citoid
ZoteroTranslationServer


008 * CCBot GPTBot ChatGPT-User anthropic-ai cohere-ai Google-Extended PerplexityBot RepoLookupBot AwarioSmartBot PerplexityAI


* ClaudeBot Amazonbot facebookexternalhit


*
msnbot
AddSearchBot
AI2Bot
Ai2Bot-DeepResearchEval
Ai2Bot-Dolma
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-extended
atlassian-bot
AutoRAG
AwarioSmartBot
AzureAI-SearchBot
Big Sur AI
bigsur.ai
Brandwatch
Bravebot
Brightbot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
Datenbank-Crawler
DeepSeekBot
Devin
Diffbot
Direqt Anomura
DuckAssistBot
Echobot Bot
ExaBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
GPTBot
iAsk
iAskBot
iaskspider
ICC Crawler
ICC-Crawler
ImageSiftBot
imageSpider
Kangaroo Bot
Kangaroo-Bot
KlaviyoAIBot
Kunato
laion-huggingface-processor
LCC
LINER Bot
LinerBot
LinkupBot
Manus-User
meta-externalagent
meta-externalfetcher
meta-webindexer
mistral.ai
MistralAI-User
netEstate Imprint Crawler
NovaAct
Novellum
Novellum AI Crawl
OAI-SearchBot
omgili
opencode
PanguBot
PeopleInc-DCipher-Scraper/1.1.0
Perplexity-User
PerplexityBot
PetalBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
SeekrBot
SemrushBot-OCOB
SemrushBotSwa
ShapBot
Spider
TavilyBot
TerraCotta
Timpibot
Trae
TwinAgent
Velen Crawler
VelenPublicWebCrawler
Webzio-Extended


Ai2Bot
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
archive.org
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
cohere-ai
Diffbot
DOC
Download Ninja
DuckAssistBot
FacebookBot
Facebookexternalhit
Fetch
Firecrawl
FirecrawlAgent
FriendlyCrawler
GPTBot
GrokApp
HMSE_Robot
HTTrack
ia_archiver
ICC-Crawler
ImagesiftBot
img2dataset
k2spider
larbin
libwww
linko
Meta-ExternalAgent
Meta-ExternalFetcher
Meta-WebIndexer
Microsoft.URL.Control
MistralAI-User
MSIECrawler
NPBot
OAI-SearchBot
Offline Explorer
omgili
OmigiliBot
PerplexityBot
PetalBot
Scrapy
Seekr
ShapBot
sitecheck.internetseer.com
SiteSnagger
Teleport
TeleportPro
TikTokSpider
Timpibot
UbiCrawler
VelenPublicWebCrawler
WebCopier
WebReaper
WebStripper
Webzio-Extended
WebZIP
wget
Xenu
YouBot
Zao
Zealbot
ZyBORG
AdsBot-Google
Mediapartners-Google
Googlebot
*


Googlebot-Image Mediapartners-Google* Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended meta-webindexer meta-externalagent meta-externalfetcher ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User cohere-training-data-crawler cohere-ai CCBot PanguBot PetalBot Bytespider Diffbot DuckAssistBot MistralAI-User Timpibot Webzio Webzio-Extended archive.org_bot ia_archiver ia_archiver-web.archive.org heritrix anthropic-ai Claude-Web FacebookBot Omgilibot YouBot Amazonbot *


AdsBot-Google
AdsBot-Google-Mobile
Mediapartners-Google
adidxbot
AdIdxBot
OAI-AdsBot
AmazonAdBot
ChatGPT-User
Claude-User
Perplexity-User
Amzn-User
Google-Agent
meta-externalads
facebookexternalhit
FacebookBot
Slackbot-LinkExpanding
Slack-ImgProxy
Google-Read-Aloud
AudigentAdBot
Criteo
CriteoBot
Discordbot
GoogleMessages
Google-NotebookLM
Google-PageRenderer
Google-Adwords-Instant
Google-Adwords-Instant-Mobile
Google-Display-Ads-Bot
LinkedInBot
meta-externalfetcher
Pinterestbot
redditbot
TaboolaBot
TelegramBot
Twitterbot
Zemanta
yahoo-ad-monitoring
Bluesky
Cardyb
TikTokSpider
Amazon-Advertising-ad-standards-bot
outbrain
Quantcastbot
Google-Pinpoint
GoogleDocs
SkypeUriPreview
Yahoo Link Preview
AddThis
bitlybot
BufferLinkPreviewBot
Snapchat
adbeat-bot
AdkernelTopicCrawler
*


* Google-Extended anthropic-ai Twitterbot GPTBot ChatGPT-User CCBot Applebot-Extended


UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
ClaudeBot
AmazonAdBot
*


* OAI-SearchBot ChatGPT-User GPTBot Perplexity-User PerplexityBot Claude-User ClaudeBot


Googlebot
Google-Extended
Bingbot
DuckDuckBot
Baiduspider
360Spider
Sogouspider
Yisouspider
ByteSpider
ToutiaoSpider
Sogou web spider
Sogou wap spider
Sogou inst spider
Sogou blog
Sogou News Spider
Sogou Orion Spider
Sosospider
PerplexityBot
Perplexity-User
OAI-SearchBot
GPTBot
ChatGPT-User
Sogou spider2
HaosouSpider
HaosoSpider
deepseekbot


* Googlebot msnbot bingbot Twitterbot Mediapartners-Google Google-Extended ScoutJet Yandex baiduspider DuckDuckBot BomboraBot grapeshot ccbot GPTBot Bard Jasper YouChat Perplexity Claude Claude-Web Neeva Copy.ai Writesonic Anyword AI21 meta-externalagent


* Twitterbot LinkedInBot GPTBot ChatGPT-User CCBot Google-Extended PerplexityBot AI2Bot Amazonbot anthropic-ai ClaudeBot Bytespider cohere-training-data-crawler cohere-ai Diffbot iAskBot/1.0 meta-externalagent MistralAI-User omgili TimpiBot YouBot


*
AmazonAdBot
AI2Bot
Bytespider
CCBot
Cohere-ai
Diffbot
FirecrawlAgent
ImagesiftBot
Kangaroo Bot
Meta-ExternalAgent
Omgilibot
PanguBot
Timpibot
Webzio-Extended
HTTrack
Nutch
Scrapy


GPTBot
ClaudeBot
Claude-User
Claude-SearchBot
CCBot
Google-Extended
Applebot-Extended
Facebookbot
Meta-ExternalAgent
Meta-ExternalFetcher
diffbot
PerplexityBot
Perplexity‑User
Omgili
Omgilibot
webzio-extended
ImagesiftBot
Bytespider
TikTokSpider
Amazonbot
Youbot
SemrushBot-OCOB
Petalbot
VelenPublicWebCrawler
TurnitinBot
Timpibot
OAI-SearchBot
ICC-Crawler
AI2Bot
AI2Bot-Dolma
DataForSeoBot
AwarioBot
AwarioSmartBot
AwarioRssBot
Google-CloudVertexBot
PanguBot
Kangaroo Bot
Sentibot
img2dataset
Meltwater
Seekr
peer39_crawler
cohere-ai
cohere-training-data-crawler
DuckAssistBot
Scrapy
Cotoyogi
aiHitBot
Factset_spyderbot
FirecrawlAgent
PetalBot
AspiegelBot
AhrefsBot
SemrushBot
DotBot
MauiBot
MJ12bot
proximic
*


008
*
AI2Bot
anthropic-ai
Claude-Web
ClaudeBot
Applebot-Extended
Bytespider
TikTokSpider
GPTBot
cohere-ai
cohere-training-data-crawler
CCBot
Cotoyogi
Ai2Bot-Dolma
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Google-Extended
PanguBot
ICC-Crawler
img2dataset
imgproxy
ISSCyberRiskCrawler
Kangaroo Bot
FacebookBot
meta-externalagent
Crawlspace
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
VelenPublicWebCrawler
omgili
omgilibot
Webzio-Extended
YouBot
aiHitBot
Brightbot 1.0
ImagesiftBot


Amazonbot
Applebot-Extended
Brightbot
ClaudeBot
Google-Extended
GPTBot
PerplexityBot
PetalBot
Scrapy
uptimerobot
viberbot
YaK
Yandex
Yeti
Applebot
Bingbot
Discordbot
DuckDuckBot
facebookexternalhit
Googlebot
Googlebot-Image
Googlebot-Video
Googlebot-News
Google-InspectionTool
LinkedInBot
msnbot
Pinterestbot
Screaming Frog SEO Spider
seznambot
Slurp
teoma
TelegramBot
Twitterbot
*


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* GPTBot ChatGPT-User CCBot anthropic-ai TextBulkerBot


Mediapartners-Google
Opebot-v (https://www.1plusx.com)
AmazonAdBot
*


*
ia_archiver
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
Slurp
Maxthon
CNCDialer
Googlebot
Googlebot-News
Googlebot-Image
Googlebot-Video


OAI-SearchBot OAI-AdsBot * AdsBot-Google


* MediaPartners-Google Updownerbot 008 bingbot msnbot


* Mediapartners-Google Googlebot-News grapeshot AmazonAdBot ClaudeBot anthropic-ai GPTBot ChatGPT-User Google-Extended PerplexityBot CCBot AdsBot-Google


Ultraseek
AhrefsBot
Algolia Crawler
*
Googlebot


* Mediapartners-Google grapeshot proximic AmazonAdBot


* GPTBot ClaudeBot Google-Extended PerplexityBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* AmazonAdBot CCBot ChatGPT-User GPTBot Google-Extended ICC-Crawler anthropic-ai Claude-Web cohere-ai omgili omgilibot


mauibot
semrushbot
ahrefsbot
blexbot
seo spider
Googlebot
*


Baiduspider Baiduspider-video Baiduspider-image Yandex YandexBot SemrushBot MJ12bot ZoominfoBot AhrefsBot *


* Googlebot AdsBot-Google AdsBot-Google-Mobile AdsBot-Google-Mobile-Apps Googlebot-Image Googlebot-Video Applebot ClarityBot bingbot MSNBot-Media


* facebookexternalhit LinkedInBot Twitterbot GPTBot


Yandex
YandexDirect
Googlebot
AdsBot-Google
AdsBot-Google-Mobile
LCC
ltx71 - (http://ltx71.com/)
bingbot
GPTBot
Claude-Web
ClaudeBot
ChatGPT-User
Google-Extended
FacebookBot
OAI-SearchBot
OAI-Fetcher
Anthropic-ai
CCBot
PerplexityBot
*


AddSearchBot
AI2Bot
Ai2Bot-DeepResearchEval
Ai2Bot-Dolma
AmazonBuyForMe
Amazonbot
Amzn-SearchBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-extended
atlassian-bot
AutoRAG
AwarioSmartBot
AzureAI-SearchBot
Big Sur AI
bigsur.ai
Brandwatch
Bravebot
Brightbot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
Datenbank-Crawler
DeepSeekBot
Devin
Diffbot
Direqt Anomura
DuckAssistBot
Echobot Bot
ExaBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
GPTBot
iAsk
iAskBot
iaskspider
ICC Crawler
ICC-Crawler
ImageSiftBot
imageSpider
Kangaroo Bot
Kangaroo-Bot
KlaviyoAIBot
Kunato
laion-huggingface-processor
LCC
LinerBot
LinkupBot
LINER Bot
Manus-User
meta-externalagent
meta-externalfetcher
meta-webindexer
mistral.ai
MistralAI-User
netEstate Imprint Crawler
Novellum AI Crawl
Novellum
NovaAct
OAI-SearchBot
omgili
opencode
PanguBot
Perplexity-User
PerplexityBot
PeopleInc-DCipher-Scraper/1.1.0
PetalBot
Poggio-Citations
QualifiedBot
SeekrBot
SemrushBot-OCOB
SemrushBotSwa
ShapBot
SBIntuitionsBot
Spider
TerraCotta
Timpibot
TavilyBot
Trae
TwinAgent
Velen Crawler
VelenPublicWebCrawler
Webzio-Extended
Wrtn
YouBot


Googlebot googlebot-image googlebot-mobile AdsBot-Google


Googlebot Googlebot-Image Googlebot-Video Googlebot-News Google-InspectionTool GoogleOther GoogleOther-Image GoogleOther-Video AdsBot-Google AdsBot-Google-Mobile Mediapartners-Google APIs-Google Google-CloudVertexBot Google-Extended Google-read-aloud Bingbot MSNBot MSNBot-Media GPTBot ChatGPT-User OAI-SearchBot PerplexityBot ClaudeBot Claude-Web anthropic-ai Applebot Applebot-Extended FacebookBot meta-externalagent CCBot ia_archiver archive.org_bot ia_archiver-web.archive.org *


ClaudeBot Google-Extended GPTBot Meta-ExternalAgent *


blinkx blinkx_ff_spider bingbot Googlebot Google-Extended GPTBot *


Amazonbot PerplexityBot Perplexity-User ClaudeBot Claude-User GPTBot ChatGPT-User AdsBot-Google AdsBot-Google-Mobile *


* Googlebot-Image Mediapartners-Google* GPTBot ChatGPT-User OAI-SearchBot Google-Extended Applebot-Extended anthropic-ai Amazonbot ClaudeBot Claude-Web Cohere-ai PerplexityBot CCBot Omgilibot Omgili Webzio-Extended FacebookBot Bytespider


* Slurp msnbot YandexBot SeznamBot discobot dotbot yacybot


MJ12bot Googlebot Mediapartners-Google AdsBot-Google AdsBot-Google-Mobile msnbot Yandex *


* google-hoteladsverifier Mediapartners-Google AdsBot-Google-Mobile AdsBot-Google AdIdxBot OAI-AdsBot SemrushBot OAI-SearchBot ChatGPT-User PerplexityBot Perplexity-User Claude-User Claude-SearchBot


Googlebot
AhrefsBot
AspiegelBot
BLEXBot
SemrushBot
Bytespider
ClaudeBot
Baiduspider-render
Sogou web spider
Sogou spider
Sogou inst spider
Sogou
newsai
Amazonbot
Applebot
Claude-SearchBot
AwarioSmartBot
AwarioRssBot
Googlebot-Image
bingbot
GoogleOther


* Nuclei WikiDo Riddler PetalBot Zoominfobot Go-http-client Node/simplecrawler CazoodleBot dotbot/1.0 Gigabot Barkrowler BLEXBot magpie-crawler


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* GPTBot CCBot SemrushBot MJ12bot BLEXBot dotbot SeekportBot AhrefsBot


mauibot
semrushbot
ahrefsbot
blexbot
seo spider
bingbot
*


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
Googlebot
Googlebot-Image
Googlebot-Video
Bingbot
msnbot
PermaBot
GPTBot
ChatGPT-User
OAI-SearchBot
Google-Extended
ClaudeBot
anthropic-ai
PerplexityBot
Bytespider
Baiduspider
Baiduspider-image
Sogou web spider
Sogou Pic Spider
360Spider
Applebot
DuckDuckBot
YandexBot
facebookexternalhit
Twitterbot
LinkedInBot


openai gptbot chatgpt-user OAl-SearchBot claudebot pangubot petalbot ChatGLM-Spider ChatGLMBot skyworkspider perplexitybot omigili cohere-ai *


* Google-Extended anthropic-ai Twitterbot GPTBot ChatGPT-User CCBot Applebot-Extended


* TurnitinBot barkrowler ClaudeBot BLEXBot MJ12bot dotbot SeekportBot AhrefsBot Amazonbot serpstatbot GPTBot AwarioRssBot AwarioSmartBot CCBot ChatGPT-User anthropic-ai Omgilibot Omgili FacebookBot Diffbot Bytespider ImagesiftBot cohere-ai Claude-User Claude-SearchBot


* OAI-SearchBot ChatGPT-User GPTBot Perplexity-User PerplexityBot Claude-User ClaudeBot GoogleOther Google-Extended


AddSearchBot
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Andibot
anthropic-ai
Applebot
Applebot-Extended
Awario
bedrockbot
bigsur.ai
Brightbot 1.0
Bytespider
CCBot
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
LinerBot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YaK
YandexAdditional
YandexAdditionalBot
YouBot


* OAI-SearchBot ChatGPT-User PerplexityBot FirecrawlAgent AndiBot ExaBot PhindBot YouBot GPTBot CCBot Google-Extended


ScreamingFrogSEOSpider
proximic
Outbrain
Yahoo Ad monitoring
SemrushBot
facebookexternalhit
ias_crawler
identrics
MeltwaterNews
links.streem.com.au
LinkedInBot
redditbot
Mediapartners-Google
Google-InspectionTool
Adsbot-Google
Storebot-Google
AdsBot-Google-Mobile
AdsBot-Google-Mobile-Apps
APIs-Google
Google-Ads-Creatives-Assistant
Google-adwords
Google-amphtml
Google-cloudvertexbot
Google-play-newsstand
Google-read-aloud
Google-Safety
Google-site-verification
Google-Structured-Data-Testing-Tool
Googleimageproxy
Applebot
AppleNewsBot
adidxbot
admantx
admantx-usbatch/3.1
AmazonAdBot
Chrome-Lighthouse
Verity
Leikibot
parselybot
peer39_crawler/1.0
pingdom
Pinterestbot
PubMatic Crawler Bot
Mobian
Slack-ImgProxy
Slackbot
Slackbot-LinkExpanding
thetradedesk
Uptimebot
yahoo-linkpreview
Googlebot
Googlebot-Image
Googlebot-News
Googlebot-Video
GoogleOther
GoogleOther-Image
GoogleOther-Video
Bingbot
bingpreview
MicrosoftPreview
DuckDuckBot
Slurp
yahoo
Google-Extended
*


*
Googlebot
Googlebot-Image
Googlebot-Video
Googlebot-News
Google-InspectionTool
GoogleOther
APIs-Google
Mediapartners-Google
DuckDuckBot
Slurp
Twitterbot
Bingbot
adidxbot
MicrosoftPreview
Applebot
AdsBot-Google
AmazonAdBot
facebookexternalhit
Yahoo Ad Monitoring
parse.ly scraper
Screaming Frog SEO Spider
SemrushBot
Botify
OnCrawl
Chrome-Lighthouse
Slackbot-LinkExpanding
Slack-ImgProxy
Slackbot
Snap URL Preview Service
proximic
GumGum-Bot
TTD-Content
Pubmatic
ias_crawler
outbrain
Leikibot
weborama-fetcher
AHC/2.1
SirdataBot
peer39_crawler/1.0
EchoboxBot/1.0
CriteoBot/0.1
AdDefend Site Context Crawler/1.0
SmartologyBot
Dianomi
ChatGPT-User
GPTBot
OAI-SearchBot
snews


* daumoa Sosospider rogerbot semetrical Googlebot-News AhrefsBot GPTBot OAI-SearchBot Applebot-Extended PerplexityBot Perplexity-ai CCBot YouBot ClaudeBot anthropic-ai Claude-Web grapeshot bingbot AmazonAdBot ozone Meta-ExternalAgent


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent Twitterbot dotbot


Googlebot Bingbot Twitterbot facebookexternalhit *


* adsbot-google AccessibleWebBot Nutch AhrefsBot AhrefsSiteAudit MJ12bot Pinterest


* Baiduspider Googlebot bingbot MSNBot 360Spider


facebookexternalhit
*
Yandex
Googlebot
Googlebot-Image
Googlebot-News
Googlebot-Video
Google-Extended
Googlebot/2.1
bingbot
Bingbot/2.0
ChatGPT-User
GPTBot
GPTBot/1.1
GPTBot/1.2
OAI-SearchBot
Meta-ExternalFetcher
Meta-ExternalAgent
FacebookBot
ClaudeBot
Claude-Web
anthropic-ai
Claude-User
Claude-SearchBot
Applebot
Applebot-Extended
Amazonbot
NovaAct
Bytespider
CCBot
Perplexity-User
PerplexityBot
DuckAssistBot
cohere-ai
cohere-training-data-crawler
AI2Bot
YouBot
MistralAI-User
Kangaroo Bot


Mediapartners-Google* Nutch Slurp AmazonAdbot Amazonbot Amzn-SearchBot Amzn-User amazon-kendra amazon-QBusiness bedrockbot aws-quick-on-behalf-of- AmazonProductDiscoverybot AmazonBuyForMe AmazonSellerInitiatedListing NovaAct * GPTBot Applebot-Extended anthropic-ai Bytespider CCBot ClaudeBot Diffbot FacebookBot Meta-ExternalAgent omgili Timpibot Webzio-Extended DDM-DCipher/1.0.7


* meta-externalagent Meta-ExternalAgent AI2Bot Diffbot Omgili Webzio-Extended


* anthropic-ai cohere-ai omgili omgilibot Applebot-Extended Bytespider CCBot ChatGPT-User ChatGPT DeepSeek DeepSeekBot GPTBot OAI-SearchBot ClaudeBot Claude-Web Diffbot FacebookBot Google-Extended Meta-ExternalAgent Meta-ExternalFetcher PerplexityBot Timpibot Webzio-Extended YouBot cXensebot


* Google-Extended GPTBot ChatGPT-User CCBot PerplexityBot anthropic-ai Claude-Web ClaudeBot Amazonbot Omgilibot FacebookBot Applebot Bytespider Diffbot ImagesiftBot Omgili YouBot


Amazonbot
Applebot
GPTBot
A6-Indexer
AhrefsBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot-Extended
Baiduspider
betaBot
BLEXBot
botify
brands-bot-logo
Brightbot
Bytespider
calculon
CCBot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
collection@infegy.com
Cotoyogi
Crawl4AI
CrunchBot
deepcrawl
DeepSeekBot
Diffbot
DirBuster
DTS
DuckAssistBot
EasouSpider
FacebookBot
FirecrawlAgent
FriendlyCrawler
GlutenCrawler
Google-CloudVertexBot
Google-Extended
GrapeshotCrawler
iaskspider
ICC-Crawler
ImagesiftBot
img2dataset
JackCrawl
Jetslide
linkapedia_bot
Mediatoolkitbot
MegaIndex
meta-externalagent
meta-externalfetcher
Microsoft BITS
Microsoft URL Control
Microsoft-WebDAV-MiniRedir
MistralAI-User
news-please
Novellum
NTENTbot
Nutch
omgili
Omgilibot
OpenLinkProfiler
Perplexity-User
PerplexityBot
PetalBot
Primalbot
python-requests
Re-Animator
Riddler
SafeDNSBot
SBIntuitionsBot
Scrapy
SearchAtlas
SemanticScholarBot
SemrushBot
SentiBot
Sogou
Spider-Pig
sqlmap
StorygizeBot
Test Certificate Info
TikTokSpider
Timpibot
Turnitin
uipbot
vBSEO
VegeBot
VelenPublicWebCrawler
Web Link Validator
WebCruiser


* SemrushBot SemrushBot-SA SemrushBot-BA SemrushBot-SI SemrushBot-SWA SemrushBot-CT SemrushBot-BM SemrushBot-SEOAB Bytespider PetalBot


* ia_archiver OmniExplorer_Bot Mediapartners-Google Baiduspider GPTBot Google-Extended Amazonbot anthropic-ai GoogleOther Claude-Web Perplexity Cohere cohere-ai Applebot-Extended Google-CloudVertexBot CCBot Omgilibot Omgili FacebookBot Bytespider Diffbot Youbot FriendlyCrawler img2dataset


*
Googlebot
Googlebot-Image
Googlebot-News
Googlebot-Video
GoogleOther
Google-InspectionTool
Storebot-Google
Bingbot
360Spider
Baiduspider
coccocbot
Daum
DuckDuckBot
Ecosia_bot
MojeekBot
Yeti
SeznamBot
Sogou web spider
Yahoo! Slurp
YandexAccessibilityBot
YandexMobileBot
Yandex
Amazonbot
ClaudeBot
Claude-User
Claude-SearchBot
CCBot
Google-Extended
FacebookBot
MistralAI-User
GPTBot
ChatGPT-User
OAI-SearchBot
PerplexityBot
Perplexity-User
archive.org_bot
AdsBot-Google
AdsBot-Google-Mobile
Chrome-Lighthouse
facebookexternalhit
Google-Read-Aloud
Google-Safety
Hatena
linkedinbot
Mediapartners-Google
Pinterestbot
SchemaBot
SiteAuditBot
SiteimproveBot
Slackbot-LinkExpanding
Telegram
Twitterbot


*
W3C-checklink
Googlebot
Yahoo! Slurp
bingbot
usasearch


* Yandex AdsBot-Google facebookexternalhit Twitterbot


Baiduspider
baiduspider
Baiduspider-image
Baiduspider-video
Baiduspider-news
Baiduspider-favo
Baiduspider-cpro
Baiduspider-ads
Sosospider
sogou spider
Sogou News Spider
Sogou inst spider
Sogou spider2
Sogou blog
Sogou Orion spider
Sogou web spider
YodaoBot
Jike Spider
360Spider
360spider-image
360spider-video
HaosouSpider
YisouSpider
EasouSpider
ToutiaoSpider
Yahoo! Slurp China
EtaoSpider
yaanibot
meta-externalagent
Meta-ExternalAgent
FacebookBot
Amazonbot
Bytespider
PetalBot
PanguBot
ChatGLM-Spider
CCBot
cohere-ai
cohere-training-data-crawler
Diffbot
AI2Bot
AI2Bot-Dolma
ImagesiftBot
img2dataset
omgili
Omgilibot
Webzio-Extended
YouBot
Timpibot
Kangaroo Bot
Sidetrade indexer bot
VelenPublicWebCrawler
iaskspider
iaskspider/2.0
360AISpider
TouTiaoSpider
MoonshotAI
Kimi-Spider
DeepSeekBot
DeepSeek-Crawler
TencentBot
BaichuanBot
BaichuanAI
SparkBot
xfspider
SkyworkBot
MiniMax-Spider
QwenSpider
DashScope
WenxinSpider
ERNIE-Bot
SenseBot
SenseChat
YuewenAI
YiBot
01ai
SemrushBot
SemrushBot-BA
SemrushBot-SI
SemrushBot-SWA
SemrushBot-CT
SemrushBot-OCOB
SemrushBot-FT
SplitSignalBot
AhrefsBot
MJ12bot
DotBot
Rogerbot
BLEXBot
DataForSeoBot
Screaming Frog SEO Spider
SearchmetricsBot
SeobilityBot
SeekportBot
SeekrBot
SEOkicks-Robot
SEOkicks
SEOstats
SeoSiteCheckup
SEOdiver Bot


*
YouBot
Webzio-Extended
VelenPublicWebCrawler
Timpibot
Sidetrade indexer bot
Scrapy
PetalBot
PerplexityBot
omgilibot
omgili
OAI-SearchBot
Meta-ExternalFetcher
Meta-ExternalAgent
Kangaroo Bot
ISSCyberRiskCrawler
img2dataset
ImagesiftBot
ICC-Crawler
iaskspider/2.0
GPTBot
GoogleOther-Video
GoogleOther-Image
GoogleOther
Google-Extended
FriendlyCrawler
facebookexternalhit
FacebookBot
Diffbot
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
cohere-ai
ClaudeBot


* GPTBot OAI-SearchBot ChatGPT-User PerplexityBot ClaudeBot Claude-SearchBot Google-Extended


* AmazonAdBot bytespider mistralai cohere ai2bot youbot omgili diffbot kangaroo img2dataset amazonbot amazon-qbusiness meta-externalagent meta-webindexer


* Googlebot msnbot bingbot Twitterbot Mediapartners-Google Google-Extended ScoutJet Yandex baiduspider DuckDuckBot BomboraBot grapeshot ccbot GPTBot Bard Jasper YouChat Perplexity Claude Claude-Web Neeva Copy.ai Writesonic Anyword AI21 meta-externalagent


amazon-kendra GPTBot ChatGPT-User PerplexityBot Perplexity-User ClaudeBot Claude-User Claude-SearchBot OAI-SearchBot Google-Extended MistralAI-User cohere-ai * PetalBot


* OAI-SearchBot ChatGPT-User PerplexityBot FirecrawlAgent AndiBot ExaBot PhindBot YouBot GPTBot CCBot Google-Extended Googlebot Bingbot


* Twitterbot facebookexternalhit Slackbot-LinkExpanding


*
msnbot
MauiBot (crawler.feedback+wc@gmail.com)


*
AddSearchBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
amazon-kendra
Amazonbot
AmazonBuyForMe
Andibot
Anomura
anthropic-ai
atlassian-bot
Awario
bedrockbot
bigsur.ai
Bravebot
Brightbot 1.0
BuddyBot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Crawlspace
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
Google-Firebase
Google-NotebookLM
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iAskBot
iaskspider
iaskspider/2.0
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
LAIONDownloader
LCC
LinerBot
Linguee Bot
LinkupBot
Manus-User
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
meta-webindexer
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NotebookLM
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot


*
AmazonAdBot
Amazon-Advertising-ad-standards-bot
008
AdsTxtCrawler
Ai2Bot-Dolma
Amazonbot
Applebot-Extended
Bytespider
CCBot
ChatGLM-Spider
ClaudeBot
CloudVertexBot
cohere-training-data-crawler
Cotoyogi
Dataprovider.com
Datenbank Crawler
dcrawl
FacebookBot
Google-Extended
GoogleOther
GPTBot
HelloworkJobPostingBot
HTTrack
HTTrack 3.0
ICC-Crawler
imageSpider
IndeedJobBot
Kangaroo Bot
KrawlerBot
laion-huggingface-processor
LCC
meta-externalagent
MetaInspector
MetaJobBot
netEstate Imprint Crawler
newspaper
Nutch
nyt_scraping
Offline Explorer
omgili
OpenindexSpider
PanguBot
Potions
SBIntuitionsBot
Scrapy
ServerHunterSpider
Spider
StatsDroneBot
Timpibot
VelenPublicWebCrawler
webzio-extended
A360-Search
aa
ActiveComply
Adagiobot
adbeat_bot
adidxbot
AdkernelTopicCrawler
admantx
AdsBot-Google
AdsBot-Google-Mobile
Ahrefs Site Audit
AhrefsBot
AhrefsSiteAudit
aiHitBot
Amazing-SearchBot
AmazingTest-SearchBot
Amazon AdBot
AmazonProductDiscovery
AndersPinkBot
ArchiveBot
Assaybot
Attracta
AudigentAdBot
Audisto Crawler
Automattic Analytics Crawler
Awario
AwarioBot
AwarioSmartBot
Barkrowler
BigUpDataBot
Bing Ads
BitSightBot
Blackboard
BLEXBot
BomboraBot
botify
BrandVerity
BrightEdge Bot
BrightEdge Crawler
Buck
CaliberBot
Caliperbot
channable
CheckMarkNetwork
Cincraw
ClarityBot
Clearscopebot
Clickagy Intelligence Bot


* AdsBot-Google AdsBot-Google-Mobile SemrushBot Baiduspider GPTBot OAI-SearchBot ChatGPT-User Google-Extended Gemini-Deep-Research ClaudeBot Claude-SearchBot Claude-User GrokBot PerplexityBot Perplexity-User Applebot-Extended CCBot


Amazonbot Applebot-Extended Brightbot ClaudeBot Google-Extended GPTBot PerplexityBot PetalBot Scrapy uptimerobot viberbot YaK Yandex Yeti facebookexternalhit meta-externalads *


* Mediapartners-Google NewsNow CCBot TurnitinBot PetalBot MoodleBot FacebookBot Bytespider Mojeek JenkersBot Seekr YouBot Arquivo-web-crawler coccocbot-web SeznamBot PerplexityBot yacy anthropic-ai ClaudeBot Claude-SearchBot Claude-User AwarioRssBot AwarioSmartBot SentiOne ImageSift Applebot-Extended YandexAdditional YandexAdditionalBot scalepostAI Buck meta-externalagent Amazonbot amazon-QBusiness DuckAssistBot Google-CloudVertexBot Amzn-SearchBot AhrefsBot AhrefsSiteAudit


* anthropic-ai cohere-ai omgili omgilibot Applebot-Extended Bytespider CCBot ChatGPT-User ChatGPT DeepSeek DeepSeekBot GPTBot OAI-SearchBot ClaudeBot Claude-Web Diffbot FacebookBot Google-Extended Meta-ExternalAgent Meta-ExternalFetcher PerplexityBot Timpibot Webzio-Extended YouBot cXensebot


Twitterbot/* * CCBot Bytespider Diffbot FacebookBot ImagesiftBot Scrapy Amazonbot


* grapeshot Googlebot-News ia_archiver Nutch ias_crawler ias_wombles ias-ie daumoa Sosospider rogerbot semetrical AhrefsBot GPTBot OAI-SearchBot Applebot-Extended PerplexityBot Perplexity-ai CCBot YouBot ClaudeBot anthropic-ai Claude-Web AmazonAdBot ozone


Googlebot Googlebot-Image AdsBot-Google GoogleOther *


AddSearchBot
AI2Bot
Ai2Bot-DeepResearchEval
Ai2Bot-Dolma
AmazonBuyForMe
Amazonbot
Amzn-SearchBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-extended
atlassian-bot
AutoRAG
AwarioSmartBot
AzureAI-SearchBot
Big Sur AI
bigsur.ai
Brandwatch
Bravebot
Brightbot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
Datenbank-Crawler
DeepSeekBot
Devin
Diffbot
Direqt Anomura
DuckAssistBot
Echobot Bot
ExaBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
GPTBot
iAsk
iAskBot
iaskspider
ICC Crawler
ICC-Crawler
ImageSiftBot
imageSpider
Kangaroo Bot
Kangaroo-Bot
KlaviyoAIBot
Kunato
laion-huggingface-processor
LCC
LinerBot
LinkupBot
LINER Bot
Manus-User
meta-externalagent
meta-externalfetcher
meta-webindexer
mistral.ai
MistralAI-User
netEstate Imprint Crawler
Novellum AI Crawl
Novellum
NovaAct
OAI-SearchBot
omgili
opencode
PanguBot
Perplexity-User
PerplexityBot
PeopleInc-DCipher-Scraper/1.1.0
PetalBot
Poggio-Citations
QualifiedBot
SeekrBot
SemrushBot-OCOB
SemrushBotSwa
ShapBot
SBIntuitionsBot
Spider
TerraCotta
Timpibot
TavilyBot
Trae
TwinAgent
Velen Crawler
VelenPublicWebCrawler
Webzio-Extended
Wrtn
YouBot


claritybot GPTBot Google-Extended CCBot ChatGPT-User anthropic-ai Omgilibot Omgili FacebookBot Bytespider * Applebot


* OAI-SearchBot ChatGPT-User ChatGPT-User/2.0 GPTBot anthropic-ai ClaudeBot claude-web PerplexityBot Perplexity-User Google-Extended Amazonbot Applebot Applebot-Extended FacebookBot meta-externalagent LinkedInBot DuckAssistBot Bytespider cohere-ai AI2Bot CCBot Diffbot omgili TimpiBot YouBot MistralAI-User SBIntuitionsBot GoogleOther YandexBot Googlebot bingbot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


Sogou web spider
Sogou inst spider
Sogou spider2
Sogou blog
Sogou News Spider
Sogou Orion spider
JikeSpider
Sosospider
Googlebot
MSNBot
Baiduspider
360spider
Baiduspider-image
YoudaoBot
*


* Sosospider AdsBot-Google AdsBot-Google-Mobile Googlebot GPTBot ChatGPT-User CCBot Google-Extended PerplexityBot SemanticScholarBot AhrefsBot BLEXBot DataForSeoBot SemrushBot


facebookexternalhit Facebot meta-externalagent Meta-ExternalFetcher *


* Google-AIO PerplexityBot Perplexity-User GPTBot ChatGPT-User Google-Extended CCBot anthropic-ai BlueJBot Spacecat/1.0


* googlebot-news AI2Bot Ai2Bot Amazonbot Applebot-Extended CCBot Cincraw ClaudeBot cohere-ai Diffbot FriendlyCrawler GPTBot ImagesiftBot img2dataset Meta-ExternalAgent PetalBot Semanticbot TimpiBot VelenPublicWebCrawler Yandex


ai2bot-dolma anthropic-ai applebot-extended bytespider ccbot claude-user claude-web claudebot cohere-ai diffbot duckassistbot Friendlycrawler omgili omgilibot perplexity-user perplexitybot scrapy timpibot youbot webzio-extended kangaroo-bot AmazonAdBot *


*
Google-CloudVertexBot
Google-Extended
ChatGPT-User
GPTBot
OAI-SearchBot
OAISearch
anthropic-ai
claude-web
claudebot
facebookbot
meta-externalagent
meta-externalfetcher
applebot-extended
perplexitybot
cohere-ai
youbot
amazonbot
aliyunsecbot
audigentadbot
awariorssbot
awariosmartbot
blexbot
bytespider
ccbot
dataforseobot
diffbot
duckassistbot
echoboxbot
friendlycrawler
ia_archiver
imagesiftbot
mycentralaiscraperbot
newsnow
news-please
omgili
omgilibot
webzio-extended
peer39_crawler
peer39_crawler/1.0
quillbot.com
quora-bot
scrapy
seekrbot
seznamhomepagecrawler
taragroup intelligent bot
timpibot
turnitinbot
viennatinybot
jetslide
magpie-crawler
poseidon research crawler


Crescent Internet ToolPak HTTP OLE Control v.1.0

Mozilla/4.0 (compatible; BullsEye; Windows 95)

Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)

Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)

Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)

Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)

Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)

Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)


AmazonAdBot * NewsNow grapeshot Googlebot


* Googlebot AdsBot-Google AdsBot-Google-Mobile AdsBot-Google-Mobile-Apps Googlebot-Image Googlebot-Video bingbot MSNBot-Media Applebot ClarityBot Neevabot


*
GPTBot
ChatGPT Agent
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-Web
Claude-User
Claude-SearchBot
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GoogleAgent-Mariner
Google-CloudVertexBot
CloudVertexBot
Meta-ExternalAgent
Meta-ExternalFetcher
meta-externalagent
meta-externalfetcher
PerplexityBot
Perplexity-User
Applebot
Applebot-Extended
MistralAI-User
MistralAI-User/1.0
cohere-ai
cohere-training-data-crawler
YouBot
PhindBot
Amazonbot
Diffbot
Bytespider
CCBot


* GPTBot ChatGPT-User ClaudeBot Claude-Web PerplexityBot Google-Extended CCBot OAI-SearchBot


AdsBot-Google-Mobile AdsBot-Google * GoogleOther


* anthropic-ai cohere-ai omgili omgilibot Applebot-Extended Bytespider CCBot ChatGPT-User ChatGPT DeepSeek DeepSeekBot GPTBot OAI-SearchBot ClaudeBot Claude-Web Diffbot FacebookBot Google-Extended Meta-ExternalAgent Meta-ExternalFetcher PerplexityBot Timpibot Webzio-Extended YouBot cXensebot


* AdsBot-Google-Mobile AdsBot-Google Googlebot-Image Twitterbot


facebookexternalhit Twitterbot LinkedInBot WhatsApp Slackbot Discordbot AdsBot-Google AdsBot-Google-Mobile *


* AmazonAdBot bytespider mistralai cohere ai2bot youbot omgili diffbot kangaroo img2dataset amazonbot amazon-qbusiness meta-externalagent meta-webindexer


AmazonAdBot Mediapartners-Google facebookexternalhit * Googlebot-News


*
webzip
AhrefsBot
webwhacker
websauger
webcapture
teleport
sitesnagger
net attache
httrack
applebot-extended
bytespider
ccbot
claudebot
google-extended
cohere-training-data-crawler
deepseekbot
deepseek
deepseek-crawler
diffbot
facebookbot
meta-externalagent
gptbot
grokbot
img2dataset
laiondownloader
mistralai-crawler
omgili
omgilibot
pangubot


* GPTBot OAI-SearchBot PerplexityBot ia_archiver Google-Extended ClaudeBot Claude-Web anthropic-ai CCBot FacebookBot proximic Moreover AhrefsBot Applebot-Extended GumGum Verity AwarioRssBot AwarioSmartBot Bytespider ChatGPT-User cohere-ai DataForSeoBot Diffbot FriendlyCrawler Google-CloudVertexBot ImagesiftBot magpie-crawler Meta-ExternalAgent meta-externalagent NewsNow news-please omgili omgilibot peer39_crawler peer39_crawler/1.0 Quora-Bot Scrapy TurnitinBot


* Amazonbot anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai DataForSeoBot Diffbot DuckAssistBot FacebookBot FriendlyCrawler Google-CloudVertexBot Google-Extended GPTbot ImagesiftBot magpie-crawler Meta-ExternalAgent meta-externalagent Meta-ExternalFetcher meta-externalfetcher NewsNow news-please OAI-SearchBot omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot Quora-Bot Scrapy Timpibot TurnitinBot YouBot


* facebookexternalhit Twitterbot LinkedInBot


AmazonAdBot
Googlebot-News
AdsBot-Google
*
AddSearchBot
AhrefsBot
AI2Bot
AI2Bot-DeepResearchEval
AI2Bot-Dolma
aiHitBot
AliyunSecBot
Amazonbot
AmazonBuyForMe
amazon-kendra
Amzn-SearchBot
Amzn-User
Andibot
Anthropic-ai
Anomura
archive.org_bot
Archive-It
atlassian-bot
AudigentAdBot
Awario
AwarioRssBot
AwarioSmartBot
BaiduSpider
bedrockbot
bigsur.ai
BLEXBot
Brandwatch
Bravebot
Brightbot 1.0
Bytespider
carbon-umbrella-bot/1.0
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT Agent
ChatGPT-User
Cision
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Crawlspace
DataForSeoBot
Datenbank Crawler
DeepSeekBot
Deepseek
Devin
Diffbot
DuckAssistBot
EchoboxBot
ErisBot
Factset_spyderbot
fiperbot/0.1
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Gnowit
GnowitNewsbot
Google-CloudVertexBot
Google-Extended
Google-NotebookLM
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
Grok
Heritrix
ia_archiver
iAskBot
iaskspider/2.0
IbouBot
ICC-Crawler
imageSpider
ImagesiftBot
img2dataset
Imprint (netEstate Imprint Crawler)
Info2424.info
ISSCyberRiskCrawler
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
LAIONDownloader
laion-huggingface-processor
LCC
LinerBot
Linguee Bot
linkfluence
LinkupBot
LinkupBot/1.0


* ClaudeBot GPTBot Claude-SearchBot OAI-SearchBot


Baiduspider
Baiduspider-video
Googlebot
YoudaoBot
Sogou web spider
Sogou inst spider
360Spider
bingbot
PerplexityBot
Perplexity-User
GPTBot
ChatGPT-User
OAI-SearchBot
deepseekbot
*


* facebookexternalhit Facebot Google-Extended AmazonAdBot AdsBot-Google Mediapartners-Google


Googlebot-News
*
anthropic-ai
Ai2Bot-Dolma
Applebot-Extended
Baiduspider
Bytespider
CCBot
ChatGLM-Spider
ClaudeBot
CloudVertexBot
cohere-training-data-crawler
Cotoyogi
DeepSeekBot
Diffbot
FacebookBot
Google-CloudVertexBot
Google-Extended
GPTBot
GrapeshotCrawler
img2dataset
Kangaroo Bot
KunatoCrawler
Meta-ExternalAgent
PanguBot
Perplexity-User
PerplexityBot
quillbot.com
Spider
TerraCotta
Timpibot
VelenPublicWebCrawler


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


AhrefsBot
AI2Bot
Ai2Bot-Dolma
Amazonbot
amazon-kendra-
anthropic-ai
Applebot-Extended
bedrockbot
Bytespider
CloudflareBrowserRenderingCrawler
CCBot
ChatGLM-Spider
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
cohere-ai
Cotoyogi
DeepSeekBot
Diffbot
DuckAssistBot
EchoboxBot
FacebookBot
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
Grok
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
KunatoCrawler
LinerBot
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MistralAI-User
OAI-Operator
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
Perplexity-User
PetalBot
QualifiedBot
Scrapy
Seekr
Sidetrade indexer bot
TaraGroup Intelligent Bot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
WRTNBot
YouBot
*
archive-it.org
archive.org_bot
ArchiveBot
Arquivo-web-crawler
Authory
babel
bl.uk_ldfc_bot
bl.uk_ldfc_renderbot
bne.es_bot
bnf.fr_bot
ecoresearch
ecoresearchCrawler
Europarchive.org
kb.dk_bot
Kulturarw3
LivelapBot
mediacloud
mirrorweb
MSIECrawler
nationalarchives.gov.uk
netarkivet.dk
netEstate NE Crawler
Nicecrawler
nl-israel_iaharvester2026
nlnbot
nlncrawler
PageFreezer
SafeDNS
SmarshBot
special_archiver
tkl.iis.u-tokyo.ac.jp
TurnitinBot
webarchiv.cz


GPTBot
Amazonbot
anthropic-ai
CCBot
Claude-web
ClaudeBot
cohere-ai
PerplexityBot
*
Googlebot-Image
Mediapartners-Google
flipboard
FlipboardProxy
5emeRue
5erue
adequat
adequat-systems
Alexibot
AlvinetSpider
AmiSoftware
Antenne Hatena
ApocalXExplorerBot
Argus
Ask n read
asknread.com
asterias
Augure
augure
auramundi
BackDoorBot/1.0
BizInformation
Black Hole
Bloodhound
Bloomberg
BlowFish/1.0
BotALot
BuiltBotTough
Bullseye/1.0
BunnySlippers
Cegbfeieh
CheeseBot
CherryPicker
CherryPickerElite/1.0
CherryPickerSE/1.0
Cision
coexel
ConveraCrawler
CopyRightCheck
Corporama
cosmos
Crescent
cydralspider
Digimind
DISCo Pump 3.1
DittoSpyder
dotbot
Download Ninja
EDD
ellisphere
EmailCollector
EmailSiphon
EmailWolf
EroCrawler
Europresse
Explore
eureka
ExtractorPro
FacebookBot
Factiva
Fasterfox
Fetch
Flamingo_SearchEngine
Foobot
gammaSpider
grub-client
Harvest/1.5
hloader
httplib
HTTrack
HTTrack 3.0
humanlinks
ia_archiver
ia_archiver-web.archive.org
Igentia
indexer
InfoNaviRobot
infoseek
JennyBot
Jetbot
JikeSpider
k2spider
Kantar
kbcrawl
Kenjin Spider
Knowings
larbin
leadbox
LexiBot
libWeb/clsHTTP
libwww


* Twitterbot LinkedInBot GPTBot ChatGPT-User CCBot Google-Extended PerplexityBot AI2Bot Amazonbot anthropic-ai ClaudeBot Bytespider cohere-training-data-crawler cohere-ai Diffbot iAskBot/1.0 meta-externalagent MistralAI-User omgili TimpiBot YouBot


GPTBot ChatGPT-User ClaudeBot Google-Extended Applebot-Extended anthropic-ai cohere-ai CCBot ICC-Crawler Bytespider OAI-SearchBot PerplexityBot Meta-ExternalAgent PetalBot baiduspider baiduimagespider Googlebot-Image Googlebot-Video Twitterbot *


ia_archiver Applebot-Extended CCBot cohere-ai cohere-training-data-crawler ClaudeBot GPTBot Meta-ExternalAgent


* GPTBot Google-Extended GoogleOther CCBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent YahooSeeker/CafeKelsa Yandex Mail.Ru/1.0 blp_bbot Baiduspider MauiBot cohere-ai ia_archiver ia_archiver-web.archive.org archive.org_bot heritrix Mediapartners-Google Twitterbot Facebot


Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/w4mwnpbXf3MFAbxOkJRw; +http://www.echobox.com)

Mozilla/5.0 (compatible; NewsRoom.BI/0.1; +http://www.newsroom.bi/bot.html)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36(KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; mrfCompass-Marshall/1.0)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Jukebox/1.0)


* BLEXBot AhrefsBot Amazonbot SemrushBot dotbot


GPTBot PerplexityBot Google-Extended anthropic-ai Claude-Web *


Baiduspider
Baiduspider-video
Googlebot
YoudaoBot
Sogou web spider
Sogou inst spider
360Spider
bingbot
*


MJ12bot
Mediapartners-Google*
IsraBot
Orthogaffe
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
fast
wget
grub-client
k2spider
NPBot
WebReaper
SemrushBot
*


Crescent Internet ToolPak HTTP OLE Control v.1.0


*
Googlebot
Bingbot
MSNBot
BingPreview
Slurp
Google-Extended
GPTBot
ClaudeBot
Claude-Web
anthropic-ai
PerplexityBot
TermlyBot
AhrefsBot
SemrushBot
MJ12bot
dotbot
CCBot
Bytespider
YandexBot
Amazonbot
amazon-QBusiness
Applebot-Extended
ImagesiftBot
MotoMinerBot
SentiBot
IRLbot
DF Bot 1.0
proximic
AwarioSmartBot


AddSearchBot
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Andibot
anthropic-ai
Applebot
Applebot-Extended
Awario
bedrockbot
bigsur.ai
Brightbot 1.0
Bytespider
CCBot
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
LinerBot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YaK
YandexAdditional
YandexAdditionalBot
YouBot


googlebot bingbot msnbot Twitterbot Slurp LinkedInBot DuckDuckBot SiteimproveBot-Crawler SemanticScholarBot facebookexternalhit Go-http-client *


*
Googlebot
Googlebot-Image
Googlebot-Video
Googlebot-News
Google-InspectionTool
GoogleOther
APIs-Google
Mediapartners-Google
DuckDuckBot
Slurp
Twitterbot
Bingbot
adidxbot
MicrosoftPreview
Applebot
AdsBot-Google
AmazonAdBot
facebookexternalhit
Yahoo Ad Monitoring
parse.ly scraper
Screaming Frog SEO Spider
SemrushBot
Botify
OnCrawl
Chrome-Lighthouse
Slackbot-LinkExpanding
Slack-ImgProxy
Slackbot
Snap URL Preview Service
proximic
GumGum-Bot
TTD-Content
Pubmatic
ias_crawler
outbrain
Leikibot
weborama-fetcher
AHC/2.1
SirdataBot
peer39_crawler/1.0
EchoboxBot/1.0
CriteoBot/0.1
AdDefend Site Context Crawler/1.0
SmartologyBot
Dianomi
ChatGPT-User
GPTBot
OAI-SearchBot


* GPTBot ChatGPT-User ClaudeBot Claude-Web anthropic-ai Google-Extended meta-externalagent PerplexityBot cohere-ai Bytespider


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


Googlebot Slurp msnbot ia_archiver Teoma SemrushBot Seobility Twitterbot facebookexternalhit GPTBot *


wget Megalodon ia_archiver libwww curl libwww-perl Scrapy Nutch indexpert Fess Mechanize Java CCBot OAI-SearchBot ChatGPT-User GPTBot Google-Extended GoogleOther AmazonBot AmazonAdBot anthropic-ai Claude-Web ICC-Crawler Applebot-Extended Bytespider PerplexityBot Perplexity-ai Perplexity-User ClaudeBot Claude-SearchBot Claude-User cohere-ai cohere-training-data-crawler Diffbot FacebookBot facebookexternalhit Gensparkbot magpie-crawler meta-externalagent meta-externalfetcher facebookcatalog omgili omgilibot Timpibot Webzio-Extended bingbot *


AddSearchBot
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Andibot
anthropic-ai
Applebot
Applebot-Extended
Awario
bedrockbot
bigsur.ai
Brightbot 1.0
Bytespider
CCBot
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
Google-Firebase
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
LinerBot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
meta-webindexer
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
ShapBot
Sidetrade indexer bot
TerraCotta
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YaK
YandexAdditional
YandexAdditionalBot
YouBot
*


LinkLint
W3C-checklink
gsa-crawler
008
Netsparker
*
ClaudeBot
GPTBot
CCBot
meta-externalagent
Bytespider
YisouSpider
Baiduspider
YandexBot
Applebot
Amazonbot
Sogou web spider


* ClaudeBot GPTBot FacebookBot AhrefsBot CCbot Bytespider cohere-ai Google-Extended Applebot-Extended FriendlyCrawler ChatGLM-Spider Scrapy DataForSeoBot ImagesiftBot


*
backlink-check.de
BacklinkCrawler
ExtractorPro
Fasterfox
LinkextractorPro
LinkWalker
MJ12bot
Openbot
rogerbot
searchpreview
SemrushBot
SEODAT
SEOENGBot
SEOkicks-Robot
True_Robot
URL Control
URL_Spider_Pro
xovi
GPTBot
um-IC
Google-Extended
ClaudeBot
Claude-Web
AhrefsBot
ImagesiftBot
GumGum Bot
anthropic-ai
ChatGPT-User
cohere-ai
PerplexityBot
Amazonbot
AmazonAdBot


* FacebookExternalHit Twitterbot Pinterestbot LinkedInBot TikTokSpider Storebot-Google Google-InspectionTool AmazonAdBot Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended Amazonbot meta-webindexer meta-externalagent meta-externalfetcher ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User cohere-training-data-crawler cohere-ai CCBot PanguBot PetalBot Bytespider Diffbot DuckAssistBot MistralAI-User Timpibot Webzio Webzio-Extended archive.org_bot ia_archiver ia_archiver-web.archive.org heritrix


* MJ12bot PetalBot SemrushBot Heritrix TurnitinBot Turnitin


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
Meltwater
NewsNow
Bloodhound
cydralspider
downloadexpress
gammaSpider
ObjectsSearch
Pimptrain
Raven
wapspider
WebZinger
Fasterfox
sentibot
GPTBot
omgili
omgilibot
CCBot
Google-Extended
Bytespider
anthropic-ai
Diffbot
PerplexityBot
YouBot
Applebot-Extended
ClaudeBot
Claude-Web
Meta-ExternalAgent
DeepSeekBot
DeepSeek
GoogleOther
Claude-SearchBot
Claude-User
cohere-ai
cohere-training-data-crawler
Amazonbot
Timpibot
AI2Bot
DuckAssistBot
Kangaroo Bot
PanguBot
MistralAI-User
Devin
PetalBot
bigsur.ai
CloudVertexBot
LinerBot
OAI-SearchBot
ChatGPT-User


ia_archiver SolomonoBot SemrushBot AhrefsBot AhrefsBot/5.0 MJ12bot dotbot dotbot/1.1 rogerbot linkdexbot/2.0 Genieo Baiduspider Baiduspider-video Baiduspider-image *


*
OAI-SearchBot
BLEXBot
Bloglines/3.1
cityreview
DOC
dotbot
Download Ninja
Exabot
Fetch
grapeshot
grub-client
HTTrack
Jyxobot/1
k2spider
larbin
libwww
linko
Microsoft.URL.Control
MSIECrawler
NPBot
Offline Explorer
PetalBot
proximic
psbot
SemrushBot
sitecheck.internetseer.com
SiteSnagger
Speedy
Teleport
TeleportPro
UbiCrawler
WebCopier
WebReaper
WebStripper
WebZIP
wget
Xenu
Zao
Zealbot
ZyBORG
AmazonAdBot


* Googlebot-Image Googlebot-News Flipboard Amazonbot Applebot Applebot-Extended Bytespider CCBot ChatGPT-User Claude-SearchBot Claude-User Claude-Web ClaudeBot FacebookBot GPTBot Google-Extended Meta-ExternalAgent MistralAI-User OAI-SearchBot Perplexity-User PerplexityBot anthropic-ai cohere-ai cohere-training-data-crawler AhrefsBot


*
Bytespider
DeepSeekBot
CCBot
YisouSpider
Baiduspider
Sogou web spider


Mediapartners-Google* Slurp AdsBot-Google* *


* Googlebot CCBot OAI-SearchBot ChatGPT-User GPTBot Google-Extended Google-CloudVertexBot ICC-Crawler anthropic-ai ClaudeBot Claude-Web Claude-SearchBot Claude-User Applebot-Extended cohere-ai Cohere-training-data-crawler omgili omgilibot PerplexityBot Perplexity-ai Perplexity-User FacebookBot Meta-ExternalAgent Meta-externalfetcher Bytespider Gensparkbot AmazonBot Diffbot Magpie-crawler Scrapy Timpibot Webzio-Extended SBIntuitionsBot SBIntuitions-SearchBot OpenindexSpider AI2Bot


GPTBot PerplexityBot Google-Extended anthropic-ai Claude-Web *


* Amazonbot AmazonBot anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web Claude-User Claude-SearchBot cohere-ai DataForSeoBot Diffbot FacebookBot Google-Extended GPTBot magpie-crawler Meta-ExternalAgent meta-externalagent MistralAI-User NagetBot NewsNow news-please OAI-SearchBot omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot Perplexity-User Scrapy TurnitinBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


Bingbot
msnbot
DuckDuckBot
Mediapartners-Google
Adsbot-Google
Googlebot
Googlebot-Mobile
Googlebot-Image
ia_archiver
SpeedySpider
TwitterBot
YandexAntivirus/2.0
YandexBot/3.0
YandexImageResizer/2.0
YandexImages/3.0
YandexMedia/3.0
YandexPagechecker/1.0
YandexWebmaster/2.0
YandexZakladki/3.0
Abonti
Aboundexbot
Acoon
Allloadin Favicon Bot
Blekkobot
Daum
Gobuster
semaltbot
Tarmot Gezgin
tchelebi
UkrNet Mail Proxy
*


* ADmantX AlphaBot anthropic-ai AwarioRssBot AwarioSmartBot BLEXBot Buzzbot Bytespider CCBot ChatGPT-User claritybot Claude-Web ClaudeBot cohere-ai Diffbot FacebookBot FriendlyCrawler Google-Extended GPTBot huggingface ImagesiftBot img2dataset magpie-crawler Meltwater Neevabot news-please NewsNow Nutch omgili omgilibot panscient.com Perplexity-ai PerplexityBot PetalBot PiplBot scoop.it Scrapy Seekr SentiBot SeznamBot TurnitinBot YouBot ZumBot


*
Branch Metrics API
Branch API
PiplBot
MJ12bot
CCBot
SeznamBot
Exabot
netseer
Mappy
crawler4j
Gigabot
ZoomBot
BUbiNG
Getintent Crawler
BLEXBot
trendictionbot
hyscore
magpie-crawler
RyteBot
GPTBot
ChatGPT-User


Crescent Internet ToolPak HTTP OLE Control v.1.0


* GPTBot ChatGPT-User PerplexityBot ClaudeBot Google-Extended Applebot-Extended


* AdsBot-Google-Mobile AdsBot-Google PetalBot dotbot AhrefsBot


* Googlebot Googlebot-Image YandexBot YandexImages Bingbot Applebot DuckDuckBot Baiduspider facebot Slurp


* facebookexternalhit Baiduspider GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User Claude-SearchBot Google-Extended PerplexityBot DuckAssistBot


* GPTBot Google-Extended anthropic-ai CCBot PerplexityBot YouBot


gsa-crawler Guidebot * Slurp AhrefsBot AhrefsSiteAudit


Googlebot
Bingbot
Slurp
DuckDuckBot
Baiduspider
YandexBot
Applebot
OAI-SearchBot
ChatGPT-User
PerplexityBot
Claude-User
Claude-SearchBot
claude-code
AdsBot-Google
AdsBot-Google-Mobile
Mediapartners-Google
facebookexternalhit
Twitterbot
LinkedInBot
Pinterestbot
Slackbot
Slackbot-LinkExpanding
Discordbot
TelegramBot
WhatsApp
ias_crawler
ias_wombles
ias_admantx
DoubleVerifyBot
DVBot
proximic
peer39_crawler
GrapeshotCrawler
grapeshot
GumGum
GlancelotBot
TTD-Content
PubMatic Crawler Bot
SinceraSyntheticUser
AmazonAdBot
Leikibot
SlickBot
GPTBot
ClaudeBot
anthropic-ai
Claude-Web
Google-Extended
Google-CloudVertexBot
Meta-ExternalAgent
Meta-ExternalFetcher
FacebookBot
Applebot-Extended
Amazonbot
Bytespider
CCBot
cohere-ai
Diffbot
YouBot
DuckAssistBot
Timpibot
Omgilibot
Omgili
ImagesiftBot
GrokBot
AI2Bot
FirecrawlAgent
Scrapy
*


* anthropic-ai AmazonBot Claude-Web cohere-ai ClaudeBot ChatGPT-User GPTBot CCBot Google-Extended DataForSeoBot Diffbot FacebookBot magpie-crawler NewsNow news-please omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot Scrapy TurnitinBot Applebot-Extended Meta-ExternalAgent meta-externalagent OAI-SearchBot Bytespider Bard Claude Anthropic


Bytespider DataForSeoBot Amazonbot AliyunSecBot *


AmazonAdBot * AI2Bot AI2Bot-Dolma Amazonbot AndiBot anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User ChatGPT-User/2.0 claudebot ClaudeBot Claude-Web cohere-ai Cohere-ai Diffbot DiffBot ExaBot FacebookBot Gemini-Deep-Research Google-CloudVertexBot Google-Extended GPTBot Img2dataset ImagesiftBot magpie-crawler Meta-ExternalAgent meta-externalagent Meta-ExternalFetcher OAI-SearchBot omgili Omgilibot PanguBot PerplexityBot PerplexityBot-User PetalBot PhindBot SentiBot Timpibot webzio-extended YouBot


* Googlebot-Image Twitterbot GPTBot ChatGPT-User OAI-SearchBot PerplexityBot CCBot Anthropic-ai DeepSeek DeepSeekBot Claude-Web


* Google-Extended Gemini-Deep-Research GPTBot OAI-SearchBot ChatGPT-User PerplexityBot Perplexity-User ClaudeBot Claude-User Claude-SearchBot Applebot-Extended meta-externalagent meta-webindexer Bytespider Amazonbot YandexBot AdsBot-Google Google-HotelAdsVerifier Google-HotelAdsVerifier/2.0 adsnaver Yeti


*
archive-it.org
archive.org_bot
ArchiveBot
Arquivo-web-crawler
Authory
babel
bl.uk_ldfc_bot
bl.uk_ldfc_renderbot
bne.es_bot
bnf.fr_bot
ecoresearch
ecoresearchCrawler
Europarchive.org
kb.dk_bot
Kulturarw3
LivelapBot
mediacloud
mirrorweb
MSIECrawler
nationalarchives.gov.uk
netarkivet.dk
netEstate NE Crawler
Nicecrawler
nl-israel_iaharvester2026
nlnbot
nlncrawler
PageFreezer
SafeDNS
SmarshBot
special_archiver
tkl.iis.u-tokyo.ac.jp
TurnitinBot
webarchiv.cz
WordPress.com mShots
XY-Archive-Compliance-Archiver
XY-Archive-Compliance-Crawler
AhrefsBot
AI2Bot
Ai2Bot-Dolma
Amazonbot
amazon-kendra-
anthropic-ai
Applebot-Extended
bedrockbot
Bytespider
CloudflareBrowserRenderingCrawler
CCBot
ChatGLM-Spider
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
cohere-ai
Cotoyogi
DeepSeekBot
Diffbot
DuckAssistBot
EchoboxBot
FacebookBot
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
Grok
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
KunatoCrawler
LinerBot
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MistralAI-User
OAI-Operator
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
Perplexity-User
PetalBot
QualifiedBot
Scrapy
Seekr
Sidetrade indexer bot
TaraGroup Intelligent Bot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent AhrefsBot


CCBot/2.0 Google-CloudVertexBot GoogleOther AhrefsBot SemrushBot Baiduspider BingBot Brightedge ChatGPT-User Claude-SearchBot Claude-User Exabot FeedFetcher-Google Google-extended Google-InspectionTool Google-Pinpoint Google-Site-Verification Google-speakr GoogleBot Googlebot-Image Googlebot-News Googlebot-Video GPTBot LinkedInBot OAI-SearchBot PerplexityBot SchemaBot/1.2 Twitterbot facebookexternalhit Qwantify MojeekBot PetalBot YouBot *


* google-hoteladsverifier Mediapartners-Google AdsBot-Google-Mobile AdsBot-Google AdIdxBot SemrushBot OAI-SearchBot ChatGPT-User PerplexityBot Perplexity-User Claude-User Claude-SearchBot


Linguee MJ12bot facebookexternalhit Twitterbot LinkedInBot Amazonbot Mediapartners-Google *


AmazonAdBot Google-InspectionTool ChatGPT-User GPTBot OAI-SearchBot *


GPTBot CCBot deepcrawl seerbot/1.0 Applebot-Extended * AmazonAdBot


* Mediapartners-Google Pinterest GPTBot ChatGPT-User CCBot ClaudeBot PerplexityBot Googlebot-News


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* Kantar GPTBot ChatGPT-User Google-Extended CCBot anthropic-ai PerplexityBot Applebot-Extended Omgilibot Bytespider YouBot


GPTBot PerplexityBot Google-Extended anthropic-ai Claude-Web *


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* CCBot ChatGPT-User OAI-SearchBot GPTBot Google-Extended anthropic-ai Omgilibot Omgili FacebookBot Diffbot Bytespider ImagesiftBot cohere-ai Claude-Web PerplexityBot


.ai
Agentic
AI Article Writer
AI Content Detector
AI Dungeon
AI Search Engine
AI SEO Crawler
AI Writer
AI21 Labs
AI2Bot
AIBot
AIMatrix
AISearchBot
AI Training
AITraining
Alexa
Alpha AI
AlphaAI
Amazon Bedrock
Amazon-Kendra
Amazon Lex
Amazon Comprehend
Amazon Sagemaker
Amazon Silk
Amazon Textract
AmazonBot
Amelia
AndersPinkBot
Anthropic
AnyPicker
Anyword
Aria Browse
Articoolo
Automated Writer
AwarioRssBot
AwarioSmartBot
Azure
BardBot
Brave Leo
ByteDance
Bytespider
CatBoost
CC-Crawler
CCBot
ChatGLM
Chinchilla
Claude
ClearScope
Cohere
Common Crawl
CommonCrawl
Content Harmony
Content King
Content Optimizer
Content Samurai
ContentAtScale
ContentBot
Contentedge
Conversion AI
Copilot
CopyAI
Copymatic
Copyscape
Cotoyogi
CrawlQ AI
Crawlspace
Crew AI
CrewAI
DALL-E
DataForSeoBot
DataProvider
DeepAI
DeepL
DeepMind
DeepSeek
Diffbot
Doubao AI
DuckAssistBot
FacebookBot
FacebookExternalHit
Falcon
Firecrawl
Flyriver
Frase AI
FriendlyCrawler
Gemini
Gemma
GenAI
Genspark
GLM
Goose
GPT
Grammarly
Grendizer
Grok
GT Bot
GTBot
Hemingway Editor
Hugging Face
Hypotenuse AI


* ADmantX AlphaBot anthropic-ai AwarioRssBot AwarioSmartBot BLEXBot Buzzbot Bytespider CCBot ChatGPT-User claritybot Claude-Web ClaudeBot cohere-ai Diffbot FacebookBot FriendlyCrawler Google-Extended GPTBot huggingface ImagesiftBot img2dataset magpie-crawler Meltwater Neevabot news-please NewsNow Nutch omgili omgilibot panscient.com Perplexity-ai PerplexityBot PetalBot PiplBot scoop.it Scrapy Seekr SentiBot SeznamBot TurnitinBot YouBot ZumBot Claude-SearchBot OAI-SearchBot


* anthropic-ai cohere-ai omgili omgilibot Applebot-Extended Bytespider CCBot ChatGPT-User ChatGPT DeepSeek DeepSeekBot GPTBot OAI-SearchBot ClaudeBot Claude-Web Diffbot FacebookBot Google-Extended Meta-ExternalAgent Meta-ExternalFetcher PerplexityBot Timpibot Webzio-Extended YouBot cXensebot


* GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-Web PerplexityBot Google-Extended


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com


* CCBot anthropic-ai Applebot cohere-ai Bytespider


MJ12bot
Mediapartners-Google*
IsraBot
Orthogaffe
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
fast
wget
grub-client
k2spider
NPBot
WebReaper
SemrushBot
*


* EasouSpider NaverBot NaverBot-1.0 AhrefsBot


AddSearchBot
AI2Bot
Ai2Bot-DeepResearchEval
Ai2Bot-Dolma
AmazonBuyForMe
Amazonbot
Amzn-SearchBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-extended
atlassian-bot
AutoRAG
AwarioSmartBot
AzureAI-SearchBot
Big Sur AI
bigsur.ai
Brandwatch
Bravebot
Brightbot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
Datenbank-Crawler
DeepSeekBot
Devin
Diffbot
Direqt Anomura
DuckAssistBot
Echobot Bot
ExaBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
GPTBot
iAsk
iAskBot
iaskspider
ICC Crawler
ICC-Crawler
ImageSiftBot
imageSpider
Kangaroo Bot
Kangaroo-Bot
KlaviyoAIBot
Kunato
laion-huggingface-processor
LCC
LinerBot
LinkupBot
LINER Bot
Manus-User
meta-externalagent
meta-externalfetcher
meta-webindexer
mistral.ai
MistralAI-User
netEstate Imprint Crawler
Novellum AI Crawl
Novellum
NovaAct
OAI-SearchBot
omgili
opencode
PanguBot
Perplexity-User
PerplexityBot
PeopleInc-DCipher-Scraper/1.1.0
PetalBot
Poggio-Citations
QualifiedBot
SeekrBot
SemrushBot-OCOB
SemrushBotSwa
ShapBot
SBIntuitionsBot
Spider
TerraCotta
Timpibot
TavilyBot
Trae
TwinAgent
Velen Crawler
VelenPublicWebCrawler
Webzio-Extended
Wrtn
YouBot


Mediapartners-Google
TruliaBot
Twitterbot
*
Mediapartners-Google*
Amazonbot
Applebot
GPTBot
A6-Indexer
AhrefsBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot-Extended
Baiduspider
betaBot
BLEXBot
botify
brands-bot-logo
Brightbot
Bytespider
calculon
CCBot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
collection@infegy.com
Cotoyogi
Crawl4AI
CrunchBot
deepcrawl
DeepSeekBot
Diffbot
DirBuster
DTS
DuckAssistBot
EasouSpider
FacebookBot
FirecrawlAgent
FriendlyCrawler
GlutenCrawler
Google-CloudVertexBot
Google-Extended
GrapeshotCrawler
iaskspider
ICC-Crawler
ImagesiftBot
img2dataset
JackCrawl
Jetslide
linkapedia_bot
Mediatoolkitbot
MegaIndex
meta-externalagent
meta-externalfetcher
Microsoft BITS
Microsoft URL Control
Microsoft-WebDAV-MiniRedir
MistralAI-User
news-please
Novellum
NTENTbot
Nutch
omgili
Omgilibot
OpenLinkProfiler
Perplexity-User
PerplexityBot
PetalBot
Primalbot
python-requests
Re-Animator
Riddler
SafeDNSBot
SBIntuitionsBot
Scrapy
SearchAtlas
SemanticScholarBot
SemrushBot
SentiBot
Sogou
Spider-Pig
sqlmap
StorygizeBot
Test Certificate Info
TikTokSpider
Timpibot
Turnitin
uipbot


* AmazonAdBot bytespider mistralai cohere ai2bot youbot omgili diffbot kangaroo img2dataset amazonbot amazon-qbusiness meta-externalagent meta-webindexer


AmazonAdBot Google-InspectionTool ChatGPT-User GPTBot OAI-SearchBot *


Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)


*
Rogerbot
Exabot
MJ12bot
Dotbot
Gigabot
AhrefsBot
BlackWidow
ChinaClaw
Custo
DISCo
Download Demon
eCatch
EirGrabber
EmailSiphon
EmailWolf
Express WebPictures
ExtractorPro
EyeNetIE
FlashGet
GetRight
GetWeb!
Go!Zilla
Go-Ahead-Got-It
GrabNet
Grafula
HMView
HTTrack
Image Stripper
Image Sucker
Indy Library
InterGET
Internet Ninja
JetCar
JOC Web Spider
larbin
LeechFTP
Mass Downloader
MIDown tool
Mister PiX
Navroad
NearSite
NetAnts
NetSpider
Net Vampire
NetZIP
Octopus
Offline Explorer
Offline Navigator
PageGrabber
Papa Foto
pavuk
pcBrowser
RealDownload
ReGet
SiteSnagger
SmartDownload
SuperBot
SuperHTTP
Surfbot
tAkeOut
Teleport Pro
VoidEYE
Web Image Collector
Web Sucker
WebAuto
WebCopier
WebFetch
WebGo IS
WebLeacher
WebReaper
WebSauger
Website eXtractor
Website Quester
WebStripper
WebWhacker
WebZIP
Wget
Widow
WWWOFFLE
Xaldon WebSpider
Zeus


Scooter
Test Crawler
DigExt
CW Crawler
Sosospider
Googlebot
*


* Flipboard FlipboardProxy GPTBot anthropic-ai ChatGPT-User ClaudeBot Claude-Web cohere-ai OAI-SearchBot quillbot.com PerplexityBot TurnitinBot AmazonAdBot


* Googlebot applebot PerplexityBot bingbot OAI-SearchBot GPTBot GeminiBot ClaudeBot


bingbot SemrushBot meta-externalagent facebookexternalhit *


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* Googlebot Bingbot DuckDuckBot Yandex SemrushBot


AI2Bot
Ai2Bot-Dolma
anthropic-ai
Applebot-Extended
Bytespider
CCBot
Claude-Web
ClaudeBot
cohere-ai
Diffbot
FacebookBot
FriendlyCrawler
Google-Extended
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
Meta-ExternalAgent
Meta-ExternalFetcher
omgili
omgilibot
Scrapy
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
Mediapartners-Google
*


* GoogleBot BingBot Yeti Daumoa Twitterbot GPTBot AdsBot-Google Ads-Naver GoogleOther facebookexternalhit facebookcatalog


Crescent Internet ToolPak HTTP OLE Control v.1.0


* AdsBot-Google AdsBot-Google-Mobile GoogleOther


ChatGPT-User OAI-SearchBot PerplexityBot Perplexity-User ClaudeBot GoogleOther Google-Extended * Skynet


Mozilla/4.0 (compatible; BullsEye; Windows 95)


* anthropic-ai cohere-ai omgili omgilibot Applebot-Extended Bytespider CCBot ChatGPT-User ChatGPT DeepSeek DeepSeekBot GPTBot OAI-SearchBot ClaudeBot Claude-Web Diffbot FacebookBot Google-Extended Meta-ExternalAgent Meta-ExternalFetcher PerplexityBot Timpibot Webzio-Extended YouBot cXensebot


*
AI2Bot
Ai2Bot-Dolma
aiHitBot
anthropic-ai
Applebot-Extended
Brightbot 1.0
CCBot
ChatGLM-Spider
ClaudeBot
Claude-Web
cohere-training-data-crawler
Cotoyogi
DeepSeekBot
Diffbot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Google-Extended
GPTBot
ICC-Crawler
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
laion-huggingface-processor
LAIONDownloader
Linguee Bot
Meta-ExternalAgent
meta-externalagent
omgili
omgilibot
PanguBot
SBIntuitionsBot
Scrapy
Sidetrade indexer bot
Spider
TerraCotta
TikTokSpider
Timpibot
VelenPublicWebCrawler
Webzio-Extended
webzio-extended
YandexAdditional
YandexAdditionalBot
Bytespider


Mediapartners-Google AdsBot-Google AdsBot-Google-Mobile googlebot * LinkedInBot LinkedInBot/1.0 jobrapido


* SemrushBot ThinkChaos LinkpadBot amazonbot gptbot yandex petalbot dotbot semrush ClaudeBot AwarioSmartBot DuckDuckBot INETDEX-BOT DataForSeoBot ImagesiftBot DotBot oBot SeznamBot YandexBot iaskspider SkyworkSpider meta-externalagent Claude-SearchBot AhrefsBot


AhrefsBot
SemrushBot
SemrushBot-BA
SemrushBot-SI
SemrushBot-SWA
SemrushBot-CT
SemrushBot-BM
SemrushBot-COUB
SiteAuditBot
SplitSignalBot
LinkCheck by Siteimprove.com
*


* AdsBot-Google-Mobile AdsBot-Google PetalBot dotbot AhrefsBot


* FacebookExternalHit Twitterbot Pinterestbot LinkedInBot TikTokSpider Storebot-Google Google-InspectionTool AmazonAdBot Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended Amazonbot meta-webindexer meta-externalagent meta-externalfetcher ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User cohere-training-data-crawler cohere-ai CCBot PanguBot PetalBot Bytespider Diffbot DuckAssistBot MistralAI-User Timpibot Webzio Webzio-Extended archive.org_bot ia_archiver ia_archiver-web.archive.org heritrix


Crescent Internet ToolPak HTTPOLE Control v.1.0


* omgili diffbot kangaroo img2dataset Greppr TikTokSpider YandexBot Yandex SeznamBot PetalBot Bingbot SemrushBot youbot amazonbot meta-externalagent meta-webindexer Baiduspider Baiduspider-image Baiduspider-video Baiduspider-news Baiduspider-favo AhrefsBot OAI-SearchBot facebookexternalhit Googlebot


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


* GPTBot Google-Extended Applebot-Extended CCBot Bytespider anthropic-ai ClaudeBot Claude-Web Claude-SearchBot Diffbot Meta-ExternalAgent FacebookBot YouBot Timpibot Amazonbot DeepSeek DeepSeekBot cohere-ai MistralAI-User Cloudflare-Crawl


* Yandex AdsBot-Google facebookexternalhit Twitterbot


CCBot Google-Extended GPTBot ChatGPT-User scrapy scrapybot SemrushBot sentibot SEOkicks-Robot TkBot trendkite-akashic-crawler Vagabondo WBSearchBot grapeshot iCjobs linkdexbot magpie-crawler MegaIndex MJ12bot proximic QuerySeekerSpider


Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot Cohere-ai GPTBot Google-CloudVertexBot Google-Extended Img2dataset PanguBot PetalBot SentiBot YouBot magpie-crawler meta-externalagent * ChatGPT-User ChatGPT-User/2.0 Gemini-Deep-Research OAI-SearchBot applebot PerplexityBot bingbot Yandex


Googlebot Bingbot Duckduckbot Slurp Baiduspider YandexBot AhrefsBot SemrushBot Moz MJ12Bot facebookexternalhit Twitterbot LinkedInBot Slackbot Pinterestbot GPTBot CCBot PerplexityBot Bytespider Amazonbot Applebot SiteimproveBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User Perplexity-User Google-Extended *


Alexa
Amazon-Kendra
AmazonBot
ByteDance
ByteSpider
CC-Crawler
CCBot
ChatGLM
ClearScopeBot
CommonCrawl
DataForSEOBot
Diffbot
Falcon
FlyRiver
Goose
Grammarly
Iaskspider
imagesiftBot
Img2dataset
PetalBot
Quillbot
Scrapy
Sidetrade
Sonic
TikTokSpider
Timpibot
VelenPublicWebCrawler
Swiftbot
Elastic-Crawler Elastic Cloud
Elastic-Crawler
Elastic Cloud
*


*
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Crawlspace
DeepSeek
DeepSeekBot
Diffbot
FacebookBot
FriendlyCrawler
Google-Extended
GPTBot
Grok
GrokAI
iaskspider/2.0
ICC-Crawler
img2dataset
ISSCyberRiskCrawler
Meta-ExternalAgent
omgili
omgilibot
PanguBot
PetalBot
Scrapy
Sidetrade indexer bot
Timpibot
TurnitinBot
VelenPublicWebCrawler
x.AI
xAI
YouBot


* OAI-SearchBot ChatGPT-User Claude-SearchBot Claude-User PerplexityBot Perplexity‑User


Googlebot AdsBot-Google Googlebot-Image Baiduspider CCBot GPTBot ClaudeBot ByteBot *


ai2bot-dolma anthropic-ai applebot-extended bytespider ccbot claude-user claude-web claudebot cohere-ai diffbot duckassistbot friendlycrawler omgili omgilibot perplexity-user perplexitybot scrapy timpibot youbot webzio-extended kangaroo-bot google-extended AmazonAdBot *


UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
grub-client
k2spider
NPBot
WebReaper
Irvine
360Spider
bingbot
wget
*
Applebot
Cliqzbot
Twitterbot
GPTBot


* GPTBot ChatGPT-User Google-Extended Googlebot Bingbot CCBot ClaudeBot PerplexityBot Amazonbot Applebot


*
PetalBot
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
GPTBot
CCBot
ByteDance


* GPTBot Applebot-Extended CCBot CloudVertexBot ClaudeBot FacebookBot meta-externalagent DeepseekBot Bytespider cohere-training-data-crawler PanguBot Timpibot AI2Bot Diffbot Omgili Webzio-Extended AmazonAdBot


Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)

Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)

Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)

Mozilla/4.0 (compatible; BullsEye; Windows 95)

Crescent Internet ToolPak HTTP OLE Control v.1.0


*
AddSearchBot
AI2Bot
Ai2Bot-Dolma
aiHitBot
AliyunSecBot
Amazonbot
Andibot
Applebot-Extended
ArchiveBot
Arquivo-web-crawler
AudigentAdBot
Awario
AwarioRssBot
AwarioSmartBot
bedrockbot
bigsur.ai
BLEXBot
BLP_bbot/0.1
Brightbot 1.0
BrightEdge
Bytespider
CCBot
ChatGPT-User
Claude-User
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
DataForSeoBot
Datenbank Crawler
Devin
Diffbot
Discordbot
dotbot
DuckAssistBot
Echobot Bot
EchoboxBot
europarchive.org
FacebookBot
Factset_spyderbot
Feedly
FirecrawlAgent
FriendlyCrawler
Google-CloudVertexBot
GoogleAgent-Mariner
heritrix
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Jetslide
LinkArchiver
LinerBot
magpie-crawler
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
MistralAI-User
MistralAI-User/1.0
MJ12bot
Moreover
MyCentralAIScraperBot
netEstate Imprint Crawler
NewsNow
news-please
Nicecrawler
NovaAct
omgili
omgilibot
Operator
PanguBot
Panscient
panscient.com
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Perplexity-User
Perplexity‑User
PhindBot
Poseidon Research Crawler
proximic
QualifiedBot
QuillBot
quillbot.com
Quora-Bot
SBIntuitionsBot
Scrapy
Screaming Frog SEO Spider
SeekrBot
Sidetrade indexer bot
SiteimproveBot
TaraGroup Intelligent Bot
TelegramBot
Thinkbot
TikTokSpider


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


* AmazonAdBot AI2Bot-Dolma anthropic-ai ApifyBot ApifyWebsiteContentCrawler Applebot-Extended Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler Diffbot FacebookBot FirecrawlAgent Google-Extended GoogleOther GPTBot imageSpider Meta-ExternalAgent omgili PanguBot Spider Timpibot Webzio-Extended


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


Mediapartners-Google * meta-externalagent GPTBot OAI-SearchBot ClaudeBot Claude-SearchBot Claude-User ChatGPT-User CCBot Applebot-Extended Google-Extended GoogleOther Google-CloudVertexBot Amazonbot Bytespider TikTokSpider DuckAssistBot PerplexityBot AhrefsBot SemrushBot MJ12bot DotBot DataForSeoBot MauiBot AlphaSeoBot BLEXBot SEOkicks-Robot Wotbox AspiegelBot PetalBot archive.org_bot ia_archiver


Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/w4mwnpbXf3MFAbxOkJRw; +http://www.echobox.com)

Mozilla/5.0 (compatible; NewsRoom.BI/0.1; +http://www.newsroom.bi/bot.html)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36(KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; mrfCompass-Marshall/1.0)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Jukebox/1.0)


grapeshot * Amazonbot Applebot Applebot-Extended Bytespider CCBot ChatGPT-User ClaudeBot Diffbot FacebookBot Google-Extended GPTBot Meta-ExternalAgent Meta-ExternalFetcher OAI-SearchBot omgili PerplexityBot Timpibot Webzio-Extended YouBot


* Googlebot AdsBot msnbot bingbot Slurp Mediapartners-Google rogerbot ClarityBot


GPTBot PerplexityBot bingbot BingPreview bingGPT Googlebot Google-Extended Gemini ClaudeBot * Googlebot-Mobile


AhrefsBot
AhrefsSiteAudit
YepBot
SemrushBot
SemrushBot-SA
Rogerbot
Exabot
barkrowler
MJ12bot
rogerbot
Dotbot
dotbot
DataForSeoBot
ia_archiver
magpie-crawler
MBCrawler
iubenda-radar
Gigabot
BlackWidow
ChinaClaw
Custo
DISCo
Download\ Demon
eCatch
EirGrabber
EmailSiphon
EmailWolf
Express\ WebPictures
ExtractorPro
EyeNetIE
FlashGet
GetRight
GetWeb!
Go!Zilla
Go-Ahead-Got-It
GrabNet
Grafula
HMView
HTTrack
Image\ Stripper
Image\ Sucker
Indy\ Library
InterGET
Internet\ Ninja
JetCar
JOC\ Web\ Spider
larbin
LeechFTP
Mass\ Downloader
MIDown\ tool
Mister\ PiX
Navroad
NearSite
NetAnts
NetSpider
Net\ Vampire
NetZIP
Octopus
Offline\ Explorer
Offline\ Navigator
PageGrabber
Papa\ Foto
pavuk
pcBrowser
RealDownload
ReGet
SiteSnagger
SmartDownload
SuperBot
SuperHTTP
Surfbot
tAkeOut
Teleport\ Pro
VoidEYE
Web\ Image\ Collector
Web\ Sucker
WebAuto
WebCopier
WebFetch
WebGo\ IS
WebLeacher
WebReaper
WebSauger
Website\ eXtractor
Website\ Quester
WebStripper
WebWhacker
WebZIP
Wget
Widow
WWWOFFLE
Xaldon\ WebSpider
Zeus


* EmailCollector sitecheck.internetseer.com Zealbot MSIECrawler SiteSnagger WebStripper WebCopier


*
Googlebot-News
AddSearchBot
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Andibot
anthropic-ai
Applebot
Applebot-Extended
Awario
bedrockbot
bigsur.ai
Brightbot 1.0
Bytespider
CCBot
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
LinerBot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
omgilibot
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YaK
YandexAdditional
YandexAdditionalBot
YouBot


* stress-agent fast Scooter AhrefsBot magpie-crawler CCBot GPTBot FacebookBot omgili omgilibot anthropic-ai Claude-Web ClaudeBot cohere-ai Google-Extended Bytespider PetalBot Diffbot PerplexityBot YouBot


* 008 Twitterbot ChatGPT-User GPTBot Google-Extended PerplexityBot Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot OAI-SearchBot ia_archiver archive.org_bot ia_archiver-web.archive.org Claude-SearchBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


psbot TurnitinBot NPBot-1/2.0 NPBot Applebot *


Googlebot
Mediapartners-Google
Yahoo! Slurp
bingbot
Yandex
Baiduspider
Sogou
ia_archiver
IPS-Agent
BLEXBot
Pandalytics
IonCrawl
*


Baiduspider
Sogou inst spider
Sogou spider
Googlebot
360Spider
haosouspider
bingbot
msnbot
msnbot-media
Sosospider
Sogou News Spider
Yahoo! Slurp China
Yahoo!
YoudaoBot
YodaoBot
Sogou web spider
YisouSpider
ia_archiver
EasouSpider
JikeSpider
EtaoSpider
*


* 80bot genieBot MJ12bot MLBot Nutch OmniExplorer_Bot spock BecomeBot TurnitinBot


* applebot-extended bytespider ccbot claudebot google-extended cohere-training-data-crawler deepseekbot deepseek deepseek-crawler diffbot facebookbot meta-externalagent gptbot grokbot img2dataset laiondownloader mistralai-crawler omgili omgilibot pangubot


usasearch
Mediapartners-Google*
bingbot
msnbot
IsraBot
Orthogaffe
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper
*


Yandex
Googlebot
01h4x.com
360Spider
404checker
404enemy
80legs
Abonti
Aboundex
Aboundexbot
Acunetix
ADmantX
adscanner
AdsTxtCrawlerTP
AfD-Verbotsverfahren
AIBOT
AiHitBot
Aipbot
Alexibot
ALittle Client
Alligator
AllSubmitter
AlphaBot
Anarchie
Anarchy
Anarchy99
Ankit
Anthill
Apexoo
arquivo-web-crawler
arquivo.pt
Aspiegel
ASPSeek
Asterias
Atomseobot
Attach
autoemailspider
awario.com
AwarioBot
AwarioRssBot
AwarioSmartBot
BackDoorBot
Backlink-Ceck
backlink-check
BacklinkCrawler
BackStreet
BackWeb
Badass
Bandit
Barkrowler
BatchFTP
Battleztar Bazinga
BBBike
BDCbot
BDFetch
BetaBot
Bigfoot
Bitacle
Black Hole
Blackboard
BlackWidow
BLEXBot
Blow
BlowFish
Boardreader
Bolt
BotALot
Brandprotect
Brandwatch
Buck
Buddy
BuiltBotTough
BuiltWith
Bullseye
BunnySlippers
BuzzSumo
Bytespider
cah.io.community
Calculon
CATExplorador
CazoodleBot
CCBot
Cegbfeieh
CensysInspect
check1.exe
CheeseBot
CherryPicker
CheTeam
ChinaClaw
Chlooe
Citoid
Claritybot
clark-crawler
Cliqzbot
Cloud mapping
coccocbot
Cocolyzebot
CODE87
Cogentbot
cognitiveseo


* newspaper/0.2.8 wget webzip webmirror webcopy magpie-crawler


*
Yandex
Active Cache Request
Googlebot-image
OAI-SearchBot
ChatGPT-User
GPTBot
Perplexity-User
PerplexityBot
Claude-User
ClaudeBot


*
GPTBot
ChatGPT-User
Google-Extended
CCBot
anthropic-ai
Claude-Web
Applebot-Extended
Amazonbot
Bytespider
ClaudeBot
cohere-ai
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
OAI-SearchBot
PerplexityBot
omgili
omgilibot
magpie-crawler
ProRataInc
PetalBot
cohere-training-data-crawler
Scrapy
img2dataset
DigitalOceanGenAICrawler
MistralAI-User
MistralAI-User/1.0
DuckAssistBot
Grok
GrokBot
DeepSeekBot
PanguBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
amazon-kendra
bedrockbot
Channel3Bot
ChatGLM-Spider
Cotoyogi
EchoboxBot
Echobot Bot
Gemini-Deep-Research
Google-CloudVertexBot
Google-NotebookLM
GoogleAgent-Mariner
iAskBot
iaskspider
iaskspider/2.0
ICC-Crawler
ImagesiftBot
KunatoCrawler
LinerBot
meta-webindexer
QualifiedBot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
wpbot
WRTNBot
ZanistaBot
KlaviyoAIBot


* AmazonAdBot bytespider mistralai cohere ai2bot youbot omgili diffbot kangaroo img2dataset amazonbot amazon-qbusiness meta-externalagent meta-webindexer


*
europarchive.org
GPTBot
ChatGPT-User
OAI-SearchBot
OpenAI-Operator
ClaudeBot
Claude-Web
anthropic-ai
Google-Extended
PerplexityBot
Applebot
FacebookBot
Meta-ExternalAgent
BingSearch
Grok
cohere-ai
Amazonbot
DeepSeek
DeepSeek Assistant
R1
YouBot
CCBot
Googlebot
Bingbot


Adsbot-Google
Alexabot
BingPreview
CloudflarePrefetch
FriendFeedBot
FunnelBack
Google
Google Favicon
Google-Site-Verification
Google-Sitemaps
Googlebot
Googlebot-Image
Googlebot-Mobile
Googlebot-News
Googlebot-Video
Mediapartners-Google
Pingdom
Pinterest
ScoutJet
Slurp
Spinn3r
Teoma
Twitterbot
Yandex
YandexImages
YandexVideoParser
Yeti
archive.org_bot
baiduspider
bingbot
facebookexternalhit
gsa-crawler
houzzbot
ia_archiver
msnbot
rogerbot
zombies
werewolves
*


* Bingbot MSNBot DuckDuckBot Baiduspider Slurp Googlebot Googlebot-Image Googlebot-Mobile Googlebot-News Googlebot-Video Yandexbot GPTBot ChatGPT-User anthropic-ai Claude-Web ClaudeBot Google-Extended CCBot cohere-ai Perplexity-bot Bingbot-Copilot OAI-SearchBot Facebookbot YouBot DuckAssistBot


applebot-extended
bytespider
ccbot
claudebot
google-extended
cohere-training-data-crawler
deepseekbot
deepseek
deepseek-crawler
diffbot
facebookbot
meta-externalagent
gptbot
grokbot
img2dataset
laiondownloader
mistralai-crawler
omgili
omgilibot
pangubot
ICCrawler - iCjobs
*


* AhrefsBot DotBot Mauibot MJ12bot PetalBot SemrushBot DataForSeoBot Neevabot MegaIndex Seekport Barkrowler ImagesiftBot BLEXBot SenutoBot GPTBot CCBot Google-Extended yacybot meta-externalagent meta-webindexer ClaudeBot Claude-SearchBot AI2Bot OAI-SearchBot thetradedesk carbon-umbrella-bot PerplexityBot Bytespider TerraCotta SiteSucker Applebot Brightbot Aranet SleepBot SERankingBacklinksBot AmazonProductDiscoveryBot Dragonfly


Googlebot Twitterbot GPTBot ChatGPT-User OAI-SearchBot Claude-User Perplexity-User


daumoa Ultraseek Ultraseek-www ia_archiver twiceler SEOkicks-Robot


Mediapartners-Google
Opebot-v (https://www.1plusx.com)
AmazonAdBot
*


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
Ai2Bot-Dolma
AI2Bot
anthropic-ai
Applebot-Extended
Arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
Barkrowler
Bytespider
CCBot
ClaudeBot
cohere-ai
cohere-training-data-crawler
crawler4j
dotbot
Diffbot
FacebookBot
FriendlyCrawler
GPTBot
https://hada.news
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MJ12bot
PanguBot
Scrapy
Seekr
Timpibot
VelenPublicWebCrawler
Webzio-Extended


* Claudebot ChatGPT-User ImagesiftBot Seekr CCBot Claude-Web PerplexityBot Google-Extended Google-CloudVertexBot GPTBot Diffbot Meltwater cohere-ai Omgilibot anthropic-ai Bytespider Amazonbot YouBot Meta-ExternalAgent DuckAssistBot


* AdsBot-Google-Mobile AdsBot-Google PetalBot dotbot AhrefsBot


* AhrefsBot SemrushBot DotBot BLEXBot MJ12bot PetalBot MauiBot DataForSeoBot


*
"008"
anthropic-ai
Applebot
Applebot-extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
Diffbot
FacebookBot
GPTBot
HTTrack
Nutch
Offline Explorer
omgili
Scrapy
YouBot


* GPTBot OAI-SearchBot ClaudeBot Claude-SearchBot PerplexityBot Amazonbot anthropic-ai Bytespider CCBot


AmazonAdBot AI2Bot-Dolma Amazonbot anthropic-ai ApifyBot ApifyWebsiteContentCrawler Applebot-Extended Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler Diffbot FacebookBot FirecrawlAgent Google-Extended GoogleOther GPTBot imageSpider meta-externalagent omgili PanguBot Spider Timpibot Webzio-Extended


Mediapartners-Google
Googlebot-Mobile
Googlebot
AdsBot-Google
AdsBot-Google-Mobile
Teoma
Slurp
Yeti
Yandex
NaverBot
bingbot
Applebot
Baiduspider
Sogou web spider
Opebot-v (https://www.1plusx.com)
AmazonAdBot
meta-externalagent
*


* CazoodleBot MJ12bot dotbot/1.0 Gigabot Mediapartners-Google AdsBot-Google Googlebot-Image Googlebot Bingbot Baiduspider GPTBot PerplexityBot ClaudeBot Google-Extended YandexBot Qwantify SeznamBot Yeti Slurp


* ia_archiver ByteSpider OAI-SearchBot ChatGPT-User PerplexityBot FirecrawlAgent AndiBot ExaBot PhindBot YouBot ClaudeBot DuckAssistBot Apple-Extended Baiduspider Yandex


* Twitterbot ia_archiver MauiBot chatgpt-user bytespider ccbot diffbot facebookbot google-extended gptbot omgili amazonbot applebot perplexitybot youbot 008 dataprovider-com dcrawl httrack httrack-3-0 metainspector newspaper nutch offline-explorer scrapy ahrefsbot barkrowler blexbot dataforseobot domainstatsbot dotbot hypestat linkdexbot mj12bot screaming-frog-seo-spider semrushbot semrushbot-ba semrushbot-ct semrushbot-si semrushbot-swa serpstatbot zoombot anthropic-ai claude-web claudebot cohere-ai Bytespider


* Googlebot Google-InspectionTool Mediapartners-Google


* Googlebot-News AI2Bot Amazonbot Applebot-Extended anthropic-ai Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai Diffbot DuckAssistBot FacebookBot GPTBot Meta-ExternalAgent Meta-ExternalFetcher OAI-SearchBot PerplexityBot


* slurp bingbot Googlebot Googlebot-image MSNBot-media MSNBot Netcraft Googlebot-Mobile AdsBot-Google Googlebot-Video Googlebot-News Mediapartners-Google AdsBots-Google-Mobile Rogerbot PowerMapper