* Amazonbot applebot-extended bytespider ccbot claudebot google-extended cohere-training-data-crawler deepseekbot deepseek deepseek-crawler diffbot facebookbot meta-externalagent gptbot grokbot img2dataset laiondownloader mistralai-crawler omgili omgilibot pangubot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


GPTBot ClaudeBot anthropic-ai Google-Extended Applebot-Extended Meta-ExternalAgent meta-externalagent Bytespider CCBot cohere-ai Diffbot Omgilibot Amazonbot DuckAssistBot PerplexityBot ChatGPT-User Claude-User Perplexity-User OAI-SearchBot Claude-SearchBot Applebot *


Mozilla/5.0 (compatible; Taboolabot/3.7; +http://www.taboola.com)


GPTBot ChatGPT-User Google-Extended CCBot *


Baiduspider
Baiduspider-video
Googlebot
YoudaoBot
Sogou web spider
Sogou inst spider
360Spider
bingbot
*


ia_archiver
archive.org_bot
WaybackMachine
archive.ph
archive.today
special_archiver
Google-Extended
GPTBot
ChatGPT-User
CCBot
Bytespider
PerplexityBot
ClaudeBot
Applebot-Extended
Amazonbot
AhrefsBot
SemrushBot
MJ12bot
BLEXBot
DotBot
seoscanners.net
Screaming Frog SEO Spider
DataForSeoBot
ZoominfoBot
MauiBot
Barkrowler
SEOkicks-Robot
AspiegelBot
PetalBot
Exabot
DotBot/1.0
Omgilibot
FacebookBot
Applebot
anthropic-ai
Claude-Web
Diffbot
ImagesiftBot
Omgili
YouBot


AmazonAdBot Google-InspectionTool ChatGPT-User GPTBot OAI-SearchBot *


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Mediapartners-Google OAI-SearchBot ChatGPT-User GPTBot Claude-User Claude-SearchBot ClaudeBot PerplexityBot


UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
larbin
libwww
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper
Aboundexbot
Aboundex/0.2
*


*
GPTBot
meta-externalagent
CCBot
Yandex
ImagesiftBot
Slurp
Yahoo! Slurp
Twitterbot
Applebot-Extended


*
Bingbot
Apple-Extended
ClaudeBot
FacebookBot
Google-Extended
GPTBot
Kangaroo Bot
PanguBot
ChatGPT-User
Operator
Applebot
OAI-SearchBot
PerplexityBot
Yandex
dotbot
rogerbot
moget
ichiro
NaverBot
Yeti
Baiduspider
Baiduspider-video
Baiduspider-image
sogou spider
YoudaoBot
MJ12bot
DJEcoBot
Amazonbot
PetalBot
AwarioSmartBot
AwarioRssBot
LinkedInBot
FreshpingBot
Pingdombot
coccocbot-web
coccocbot-image
sethbot
UptimeRobot
Mail.RU_Bot
heritrix
DuckAssistBot
YouBot
Meta-ExternalFetcher
Meta-ExternalAgent
Webzio-Extended
Timpibot
Omgili
Diffbot
cohere-training-data-crawler
Bytespider
CCBot
AI2Bot
NextGenSearchBot
ZoomBot
ZoominfoBot
DataForSeoBot


* AdsBot Adsbot-Google AdsBot-Google-Mobile AdsBot-Google-Mobile-Apps AdIdxBot Applebot APIs-Google baiduspider Bingbot BingPreview Caliperbot CCBot ContentKing Deepcrawl DuckDuckBot facebot facebookexternalhit/1.0 facebookexternalhit/1.1 Googlebot Googlebot-Image Googlebot-News Googlebot-Video LinkedInBot MicrosoftPreview Mediapartners-Google Neevabot PetalBot SemrushBot Slurp Siteimprove SemanticScholarBot Storebot-Google Twitterbot YandexBot


*
Mediapartners-Google
googlebot
FacebookBot
bingbot(at)microsoft.com
Bingbot
Baiduspider
YandexBot
Isposure Agent
proximic
AhrefsBot
Go-http-client
Giant Oak MN
magpie-crawler
GPTBot
ChatGPT-User
Google-Extended
CCBot


* AmazonAdBot anthropic-ai ChatGPT-User ClaudeBot Claude-Web cohere-ai GPTBot PerplexityBot Bytespider


Mozilla/4.0 (compatible; MSIE 5.5; Windows NT; PCLN NOC SiteScope)


* Yandex AdsBot-Google facebookexternalhit Twitterbot


Amazonbot anthropic-ai Applebot-Extended CCBot ChatGPT-User Claude-Web ClaudeBot cohere-ai DuckAssistBot FacebookBot Google-CloudVertexBot Google-Extended GPTBot Meta-ExternalAgent meta-externalagent PerplexityBot Quora-Bot AdsBot-Google AdsBot-Google-Mobile AdsBot-Google-Mobile-Apps *


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent dotbot Yandex sitebot MJ12bot wotbox discoverybot BLEXBot AhrefsBot XoviBot SemrushBot SemrushBot-SA


*
Algolia Crawler
facebookexternalhit
Twitterbot
Mediapartners-Google*


* ADmantX AlphaBot anthropic-ai AwarioRssBot AwarioSmartBot BLEXBot Buzzbot Bytespider CCBot ChatGPT-User claritybot Claude-Web ClaudeBot cohere-ai Diffbot FacebookBot FriendlyCrawler Google-Extended GPTBot huggingface ImagesiftBot img2dataset magpie-crawler Meltwater Neevabot news-please NewsNow Nutch omgili omgilibot panscient.com Perplexity-ai PerplexityBot PetalBot PiplBot scoop.it Scrapy Seekr SentiBot SeznamBot TurnitinBot YouBot ZumBot


Crescent Internet ToolPak HTTP OLE Control v.1.0


msnbot-media/1.1 msnbot-media/2.0b Googlebot-News scrapy Diffbot RyteBot proximic GPTBot ChatGPT-User Applebot-Extended CCBot Google-Extended *


* Google-Extended CCBot GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-User Claude-SearchBot anthropic-ai Meta-ExternalAgent FacebookBot PerplexityBot Perplexity-User Applebot-Extended Gemini-Deep-Research Amazonbot Bytespider cohere-ai YouBot Omgilibot Omgili Diffbot Googlebot YisouSpider Baiduspider YandexBot AhrefsBot PetalBot


*
CCBot
ChatGPT-User
GPTBot
Google-Extended
Google-CloudVertexBot
Applebot-Extended
anthropic-ai
ClaudeBot
Omgilibot
Omgili
FacebookBot
Diffbot
DuckAssistBot
AI2Bot
Bytespider
Kangaroo Bot
ImagesiftBot
PerplexityBot
cohere-ai
Meta-ExternalAgent
Meta-ExternalFetcher
Timpibot
Webzio-Extended
YouBot
DeepSeek


AdsBot-Google-Mobile Googlebot-Image AdsBot-Google Googlebot PetalBot YandexBot *


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


AI2Bot
Ai2Bot-Dolma
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
Diffbot
FacebookBot
FriendlyCrawler
GPTBot
Google-Extended
ICC-Crawler
ImagesiftBot
Meta-ExternalAgent
OAI-SearchBot
PerplexityBot
PetalBot
Scrapy
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot
anthropic-ai
cohere-ai
iaskspider/2.0
img2dataset
omgili
omgilibot
cohere-training-data-crawler
Kangaroo Bot
PanguBot
Amazonbot
DuckAssistBot
Meta-ExternalFetcher
Perplexity-User
Claude-SearchBot
Claude-User
Cotoyogi
Datenbank
Devin
Factset_spyderbot
Gemini-Deep-Research
MistralAI-User
netEstate
Imprint
SemrushBot-OCOB
AddSearchBot
bigsur.ai
CloudVertexBot
GoogleOther
LinerBot
meta-externalagent
QualifiedBot
Ai2Bot-DeepResearchEval
ChatGLM-Spider
DeepSeekBot
Google-NotebookLM
iAskBot
KunatoCrawler
meta-externalfetcher
meta-webindexer
webzio-extended
WRTNBot
ZanistaBot
AI2Bot-DeepResearchEval
Anomura
atlassian-bot
Bravebot
Channel3Bot
Cloudflare-AutoRAG
Crawl4AI
Google-CloudVertexBot
imageSpider
KlaviyoAIBot
laion-huggingface-processor
LCC
LinkupBot
PhindBot
Poggio-Citations
SBIntuitionsBot
TavilyBot
Amzn-SearchBot
Amzn-User
ApifyBot
ApifyWebsiteContentCrawler
Aranet-SearchBot
AzureAI-SearchBot
ExaBot
FirecrawlAgent
kagi-fetcher
TerraCotta
*


* Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot Claude-Web cohere-ai DataForSeoBot Diffbot FacebookBot FriendlyCrawler Google-Extended GPTBot Meta-ExternalAgent meta-externalagent omgili omgilibot PerplexityBot Quora-Bot Scrapy TurnitinBot


* Mediapartners-Google Googlebot Googlebot-image NaverBot Yeti Daumoa Twitterbot OAI-SearchBot ChatGPT-User Claude-SearchBot Claude-User PerplexityBot Perplexity-User Google-Extended ads-naver


* RavenCrawler rogerbot dotbot SemrushBot SiteAuditBot SplitSignalBot PowerMapper Swiftbot lyticsbot Dubbotbot PopeTech-CrawlBot PopeTech-ScanBot


* Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai Diffbot FacebookBot facebookexternalhit FriendlyCrawler Google-Extended GPTBot ICC-Crawler ImagesiftBot img2dataset meta-externalagent OAI-SearchBot Omgili Omgilibot PerplexityBot PetalBot Scrapy Timpibot VelenPublicWebCrawler YouBot


* GPTBot ChatGPT-User ClaudeBot Claude-Web PerplexityBot Applebot-Extended Bytespider CCBot cohere-ai Meta-ExternalAgent OAI-SearchBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
googlebot-news
YakazBot
trovitBot
Eyeotabot
PiplBot
CriteoBot/0.1
trendkite-akashic-crawler
GPTBot
CCBot
anthropic-ai
cohere-ai
ia_archiver
omgili
omgilibot
MJ12bot
meta-externalagent
ClaudeBot
Claude-User
Claude-SearchBot
Bytespider
PerplexityBot
Perplexity-User
Ai2Bot
Kangaroo Bot
PanguBot
Amazonbot
Applebot
Applebot-Extended
bingbot


* Googlebot Googlebot-Mobile AmazonAdBot


*
GPTBot
CCBot
anthropic-ai
Google-Extended
PetalBot
Yandex
Amazonbot
Applebot
OAI-SearchBot
PerplexityBot
QualifiedBot
ICC Crawler
Meta-ExternalAgent
Factset_spyderbot
ChatGPT-User
Perplexity-User
DuckAssistbot


Googlebot Googlebot-Image AdsBot-Google * PowerMapper Amazonbot


* Google-Extended Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler GPTBot meta-externalagent


* Googlebot-News facebookexternalhit Facebot Google-Extended Mediapartners-Google AdsBot-Google AmazonAdBot


Meta-ExternalAgent Meta-ExternalFetcher facebookexternalhit * GPTBot Claude ClaudeBot


*
grapeshot
AmazonAdBot
008 Disallow: /
AddThis.com Disallow: /
admantx Disallow: /
AhrefsBot Disallow: /
BDCbot Disallow: /
Bender Disallow: /
BIXOCRAWLER Disallow: /
bl.uk_lddc_bot Disallow: /
BLEXBot Disallow: /
BUbiNG Disallow: /
Cliqzbot Disallow: /
CNCDialer Disallow: /
crawler4j Disallow: /
CrystalSemanticsBot Disallow: /
CyberAlert Disallow: /
DigExt Disallow: /
discobot Disallow: /
discoverybot Disallow: /
dloader Disallow: /
dloader(NaverRobot) Disallow: /
DOC Disallow: /
dotbot Disallow: /
Download Ninja Disallow: /
DTS Agent Disallow: /
Exabot Disallow: /
Ezooms Disallow: /
FairShare Disallow: /
Fetch Disallow: /
Flamingo_SearchEngine Disallow: /
Genieo Disallow: /
Gigabot Disallow: /
grub-client Disallow: /
Heritrix Disallow: /
heritrix/3.3.0 Disallow: /
HTTrack Disallow: /
ia_archiver Disallow: /
integromedb Disallow: /
IstellaBot Disallow: /
JikeSpider Disallow: /
Jyxobot Disallow: /
k2spider Disallow: /
Kimengi Disallow: /
Kimengi/nineconnections.com Disallow: /
larbin Disallow: /
LexxeBot/1.0 Disallow: /
libwww Disallow: /
linko Disallow: /
Livelapbot Disallow: /
magpie-crawler Disallow: /
Maxthon Disallow: /
MetaURI Disallow: /
Microsoft.URL.Control Disallow: /
MJ12bot Disallow: /
Moreover Disallow: /
Moreoverbot Disallow: /
MSIECrawler Disallow: /
nabot Disallow: /
NaverBot Disallow: /
NerdByNature.Bot Disallow: /
netEstate NE Crawler Disallow: /
NetSeer crawler Disallow: /
Newscan Disallow: /
NextGenSearchBot Disallow: /
NPBot Disallow: /
nutch Disallow: /
Offline Explorer Disallow: /
omgilibot Disallow: /
Orthogaffe Disallow: /
PiplBot Disallow: /
Pixray-Seeker Disallow: /
proximic Disallow: /
psbot Disallow: /
QuerySeekerSpider Disallow: /
rogerbot Disallow: /
seokicks Disallow: /
SEOkicks-Robot Disallow: /
SiteBot Disallow: /
SiteBot/0.1 Disallow: /
sitecheck.internetseer.com Disallow: /
SiteSnagger Disallow: /
Slurp Disallow: /
sogou Disallow: /
Sosospider Disallow: /
spbot Disallow: /
Spinn3r Disallow: /
Teleport Disallow: /
TeleportPro Disallow: /
trendictionbot Disallow: /
trovitBot Disallow: /
TurnitinBot Disallow: /
UbiCrawler Disallow: /
uMBot-LN Disallow: /
UnisterBot Disallow: /
UniversalFeedParser Disallow: /
WBSearchBot Disallow: /
WebCopier Disallow: /
WebReaper Disallow: /


Googlebot
Mediapartners-Google
Yahoo! Slurp
bingbot
Yandex
Baiduspider
Sogou
ia_archiver
IPS-Agent
BLEXBot
Pandalytics
IonCrawl
*


CCBot GPTBot ChatGPT-User OAI-SearchBot Google-Extended ICC-Crawler ClaudeBot anthropic-ai Claude-Web PerplexityBot Perplexity-ai Applebot-Extended cohere-ai Meta-ExternalAgent Gensparkbot Bytespider omgili omgilibot cohere-training-data-crawler Diffbot FacebookBot magpie-crawler meta-externalfetcher Scrapy Timpibot Webzio-Extended AmazonBot Claude-SearchBot Claude-User AI2Bot


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


* AdsBot-Google-Mobile AdsBot-Google PetalBot dotbot AhrefsBot


*
Twitterbot
Twitterbot/1.0
Xbot
X-Twitter-Image-Crawler
Amazonbot
GPTBot
OpenAI
ChatGPT-User
CCBot
anthropic-ai
Claude-Web
PerplexityBot
Applebot-Extended
AudigentAdBot
AwarioRssBot
AwarioSmartBot
BLEXBot
Bytespider
ClaudeBot
cohere-ai
DataForSeoBot
Diffbot
DuckAssistBot
EchoboxBot
FacebookBot
FriendlyCrawler
ImagesiftBot
Jetslide
magpie-crawler
Meta-ExternalAgent
meta-externalagent
Meta-ExternalFetcher
meta-externalfetcher
Meta-WebIndexer
MistralAI-User
MyCentralAIScraperBot
NewsNow
news-please
OAI-SearchBot
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
Poseidon Research Crawler
quillbot.com
Quora-Bot
Scrapy
SeekrBot
SeznamHomepageCrawler
TaraGroup Intelligent Bot
Timpibot
TurnitinBot
ViennaTinyBot
YouBot
newspaper
Perplexity-User
docomo
yisouspider
trendictionbot
tin_eye
metaexternalagent
chatgpt_user
Claude-User
Claude-SearchBot
ias-va
ias-or
ias-ir
ias-jp
ias-sg
ias-au
ias_crawler
admantx
Webz.io
ICC-Crawler
you.com.
ChatGPT’s agent Operator.
Bing’s Copilot chat.
Grok.
archive.org_bot
archive_bot
ia_archiver
Baiduspider
baiduspider
Baiduspider-image
Baiduspider-video


* AdsBot-Google Googlebot-image Yandex Prerender CCBot ChatGPT-User GPTBot anthropic-ai Google-Extended ClaudeBot Claude-Web cohere-ai


* OAI-AdsBot OAI-SearchBot GPTBot PerplexityBot


*
Ai2Bot-Dolma
AI2Bot
anthropic-ai
Applebot-Extended
Arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
Barkrowler
Bytespider
CCBot
ClaudeBot
cohere-ai
cohere-training-data-crawler
crawler4j
dotbot
Diffbot
FacebookBot
FriendlyCrawler
GPTBot
https://hada.news
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MJ12bot
PanguBot
Scrapy
Seekr
Timpibot
VelenPublicWebCrawler
Webzio-Extended


*
GPTBot
ClaudeBot
Google-Extended
Applebot-Extended
Meta-ExternalAgent
Amazonbot
CCBot
bingbot
bingpreview
AhrefsBot
SemrushBot
Diffbot
UbiCrawler
BUbiNG
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper
rogerbot
SemrushBot-SA
BrandVerity
Bytespider
DeepSeekBot


* Applebot Googlebot bingbot msnbot Slurp Facebot baiduspider sosobot exabot Mediapartners-Google


MJ12bot
Mediapartners-Google*
IsraBot
Orthogaffe
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
fast
wget
grub-client
k2spider
NPBot
WebReaper
SemrushBot
*


Baiduspider
Sosospider
sogou spider
SemrushBot
AhrefsBot
PetalBot
*


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot GPTBot meta-externalagent anthropic-ai Applebot AwarioRssBot AwarioSmartBot ChatGPT-User Claude-Web cohere-ai DataForSeoBot Diffbot FacebookBot FriendlyCrawler img2dataset ImagesiftBot magpie-crawler Meltwater omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot PiplBot scoop.it Seekr YouBot


* GPTBot Google-Extended CCbot ClaudeBot


* facebookexternalhit Facebot Google-Extended AmazonAdBot AdsBot-Google Mediapartners-Google


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot GPTBot meta-externalagent anthropic-ai Applebot Awario* ChatGPT-User Claude* cohere-ai DataForSeoBot Diffbot FacebookBot FriendlyCrawler img2dataset ImagesiftBot magpie-crawler Meltwater NeevaBot omgili* peer39_crawler PerplexityBot PiplBot scoop.it Seekr YouBot AhrefsBot SemrushBot MJ12bot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


AmazonAdBot Google-InspectionTool ChatGPT-User GPTBot OAI-SearchBot *


*
MJ12Bot
PetalBot
AspiegelBot
AhrefsBot
SEMrushBot
DotBot
MauiBot
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot-Extended
Brightbot 1.0
Bytespider
CCBot
ChatGLM
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Crawlspace
DeepSeekBot
Diffbot
FacebookBot
FriendlyCrawler
Google-Extended
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
imgproxy
ISSCyberRiskCrawler
Kangaroo Bot
Meta-ExternalAgent
omgili
omgilibot
PanguBot
Perplexity-User
PerplexityBot
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot


* facebookexternalhit LinkedInBot Twitterbot GPTBot


Googlebot Bingbot OAI-SearchBot ChatGPT-User Google-Extended GPTBot ClaudeBot Claude-SearchBot Claude-User PerplexityBot GoogleOther BingPreview * ScrapeHero


* GPTBot OAI-SearchBot ChatGPT-User ClaudeBot anthropic-ai Claude-Web PerplexityBot Perplexity-User Google-Extended Applebot-Extended Amazonbot cohere-ai CCBot Diffbot Bytespider meta-externalagent FacebookBot adsbot-google Nutch AhrefsBot AhrefsSiteAudit MJ12bot Pinterest


* OAI-SearchBot ChatGPT-User Claude-User Claude-SearchBot Claude-Web PerplexityBot Perplexity-User DuckAssistBot Amazonbot MJ12bot


* AppleNewsBot Googlebot-News Amazonbot anthropic-ai Bytespider ChatGPT-User Claude-Web cohere-ai Diffbot FacebookBot Google-Extended ImagesiftBot NewsNow news-please OAI-SearchBot omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot PerplexityUser Scrapy AI2Bot amazon-kendra Applebot-Extended Bedrock-bot CCBot ChatGLM-Spider Claude-SearchBot Claude-User ClaudeBot Cotoyogi DeepSeek DuckAssistBot EchoboxBot Gemini-Deep-Research Google-CloudVertexBot GPTBot iAskBot ICC-Crawler Kunato LinerBot Meta-ExternalAgent Meta-ExternalFetcher mistral.ai MistralAI-User PanguBot ProRataInc QualifiedBot SeekrBot TikTokSpider VelenPublicWebCrawler WARDBot WPBot WrtnBot


Applebot-Extended anthropic-ai CCBot ChatGPT-User ClaudeBot Claude-Web FacebookBot Google-Extended GPTBot ia_archiver OAI-SearchBot omgili omgilibot PerplexityBot PerplexityUser


* GoogleBot R6_FeedFetcher Genieo genieo Pinterestbot deepcrawl


AdsBot-Google Googlebot-Image Googlebot *


*
proximic
Yandex
Baiduspider
008
AddThis.com
admantx
BDCbot
Bender
bl.uk_lddc_bot
BLEXBot
BUbiNG
CNCDialer
CrystalSemanticsBot
CyberAlert
discobot
discoverybot
DOC
DotBot
Download Ninja
Exabot
Ezooms
Fetch
Flamingo_SearchEngine
grub-client
heritrix
HTTrack
integromedb
IstellaBot
JikeSpider
Jyxobot
k2spider
larbin
LexxeBot/1.0
libwww
linko
magpie-crawler
Maxthon
MJ12bot
Moreoverbot
MSIECrawler
NerdByNature.Bot
netEstate NE Crawler
NetSeer crawler
Newscan
NextGenSearchBot
NPBot
Offline Explorer
Pixray-Seeker
psbot
rogerbot
seokicks
SiteBot
SiteBot/0.1
sitecheck.internetseer.com
SiteSnagger
sogou
Sosospider
spbot
Spinn3r
Teleport
TeleportPro
trovitBot
TurnitinBot
UbiCrawler
UnisterBot
WBSearchBot
WebCopier
WebReaper
WebStripper
WebZIP
WeSEE:Search
wget
wotbox
Yasni
Zao
Zealbot
ZyBORG
anthropic-ai
AI2Bot
Ai2Bot-Dolma
AmazonBot
ApifyBot
ApifyWebsiteContentCrawler
Applebot-Extended
Bytespider
CCBot
ChatGLM-Spider
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Diffbot
FacebookBot
FirecrawlAgent
GPTBot
iAskBot


* Nuclei WikiDo Riddler PetalBot Zoominfobot Go-http-client Node/simplecrawler CazoodleBot dotbot/1.0 Gigabot Barkrowler BLEXBot magpie-crawler


* GPTBot ChatGPT-User Google-Extended ClaudeBot anthropic-ai PerplexityBot Amazonbot Bytespider cohere-ai


Applebot-Extended
Atomic_Email_Hunter
Bender
Birubot
BlogPulseLive
BOTW Spider
Butterfly
CCBot
ChatGPT-User
Claude-SearchBot
Claude-User
ClaudeBot
Crowsnest
discobot
DoCoMo
dotbot
Exabot
expo9
Ezooms
FairShare
Feedtrace
Ginxbot
Google-Extended
gosospider
GPTBot
knowaboutBot
Meta-ExternalAgent
MetaURI
MJ12bot
MLBot
Moreoverbot
mxbot
my6sense
netseer
NHN Corp
NjuiceBot
Nutch
OAI-AdsBot
OAI-SearchBot
OneRiot
percbotspider
PerplexityBot
Perplexity-User
Postrank
proximic
Purebot
radian6
ScoutJet
semiocast
Showyoubot
SiteBot
suggybot
Summify
Tagoobot
TANNER
ThingFetcher
trendictionbot
TweetedTimes Bot
TweetmemeBot
Twiceler
Twingly Recon
twitmatic
Twitturls
Twitturly
Typhoeus
Viralheat
YodaoBot
YouBot
*


Googlebot Bingbot Applebot AhrefsBot Amazonbot Bytespider PetalBot DuckAssistBot Meta-ExternalAgent ClaudeBot GPTBot Meta-ExternalFetcher PerplexityBot


* bingbot AdsBot-Google AdIdxBot Twitterbot Bytespider CCBot ChatGLM-Spider ClaudeBot PerplexityBot anthropic-ai AmazonBot meta-externalagent GPTBot Applebot-Extended


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* GPTBot OAI-SearchBot ChatGPT-User PerplexityBot ClaudeBot anthropic-ai Applebot-Extended Google-Extended


*
DuckAssistBot
Gemini-Deep-Research
Meta-ExternalFetcher
MistralAI-User
ChatGPT-User
PerplexityBot
Perplexity‑User
Claude-User
AI2Bot
Applebot-Extended
Bytespider
CCBot
ClaudeBot
cohere-training-data-crawler
Cotoyogi
Datenbank Crawler
Diffbot
FacebookBot
Factset_spyderbot
Google-Extended
GPTBot
ICC-Crawler
Kangaroo Bot
Meta-ExternalAgent
netEstate Imprint Crawler
omgili
PanguBot
PetalBot
SemrushBot-OCOB
Timpibot
Webzio-Extended


* OAI-SearchBot CCBot PerplexityBot anthropic-ai Claude-Web ClaudeBot GPTBot ChatGPT-User


* AmazonAdBot bytespider mistralai cohere ai2bot youbot omgili diffbot kangaroo img2dataset amazonbot amazon-qbusiness meta-externalagent meta-webindexer


* applebot-extended bytespider ccbot claudebot google-extended cohere-training-data-crawler deepseekbot deepseek deepseek-crawler diffbot facebookbot meta-externalagent gptbot grokbot img2dataset laiondownloader mistralai-crawler omgili omgilibot pangubot


* GPTBot ClaudeBot Google-Extended CCBot anthropic-ai Bytespider PetalBot


Crescent Internet ToolPak HTTP OLE Control v.1.0


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


AddSearchBot
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Andibot
anthropic-ai
Applebot
Applebot-Extended
Awario
bedrockbot
bigsur.ai
Brightbot 1.0
Bytespider
CCBot
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
Google-Firebase
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
LinerBot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
meta-webindexer
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
ShapBot
Sidetrade indexer bot
TerraCotta
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YaK
YandexAdditional
YandexAdditionalBot
YouBot
*


GPTBot CCBot deepcrawl seerbot/1.0 Applebot-Extended * AmazonAdBot


*
googlebot
googlebot-image
GoogleOther
Googlebot-Video
Google-InspectionTool
AdsBot-Google
AdsBot-Google-Mobile
AdsBot-Google-Mobile-Apps
Storebot-Google
google-read-aloud
mediapartners-google
bingbot
msnbot
bingpreview
slurp
yahoo
yandex
YandexImages
baiduspider
Pinterestbot
Yeti
MojeekBot
360Spider
google-cloudvertexbot
duckduckbot
petalbot
Applebot
AppleNewsBot
flipboard
qwantbot
SeznamBot
yisouspider
proximic
admantx
thetradedesk
outbrain
ias_crawler
AmazonAdBot
pubmatic
smartologybot
parselybot
Screaming Frog SEO Spider
DeepCrawl
AhrefsBot
SemrushBot
SimilarWebBot
SISTRIX
OnCrawl
botify
Chrome-Lighthouse
DJSEOcrawler
ChatGPT-User
GPTBot
OAI-SearchBot
facebookexternalhit
facebot
twitterbot
linkedinbot
telegrambot
snapchat
newrelicpinger
siteimprovebot
uptimerobot
pingdom.com_bot_version_1.4
sentry
schema-validator
googleimageproxy
Iframely
Vocabtracker
EpvzCrawl6194680250
Citoid
ZoteroTranslationServer


*
GPTBot
PerplexityBot
Amazonbot
anthropic-ai
AwarioRssBot
AwarioSmartBot
Bytespider
CCBot
HuggingFaceBot
ChatGPT-User
ClaudeBot
Claude-Web
DataForSeoBot
Diffbot
FacebookBot
Google-Extended
Google-NotebookLM
Google-Agent
magpie-crawler
NewsNow
news-please
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
Scrapy
TurnitinBot
cohere-ai
Applebot-Extended
Meta-ExternalAgent
meta-externalagent
OAI-SearchBot
Claude-User
polymarket-news-bot
KalshiBot
polymarket-bot
OpenClaw
gold-ai-event-bridge
TradingBot
trading-pipeline
kalshi-bot
BivariateKalshiPipeline
Falcon Trading Platform
StockMarketAnalyser
FinanceApp
polyclaw-ingester
IsoxyTrader
Neo Trading Bot
MerlinChartWizard
investment-dashboard


*
Amazonbot
Googlebot-Image
Mediapartners-Google
Storebot-Google
AdsBot-Google
Baiduspider
360Spider
Yisouspider
PetalBot
Bytespider
Sogou web spider
Sogou inst spider
Robozilla
yahoo-mmcrawler
psbot
asterias
yahoo-blogs/v3.9
Yandex
BLEXBot
Nutch


* Mediapartners-Google AdsBot-Google-Mobile AdsBot-Google Slurp Baiduspider


ActiveComplyBot
CriteoBot
GumGumBot
TaraGroup Intelligent Bot
*


*
Googlebot
seznambot
SemrushBot
LinkpadBot
BLEXBot
FlipboardProxy
aiHitBot
trovitBot
grapeshot
MegaIndex
MJ12bot
Detectify
Riddler
DotBot
DISCo Pump
Wget
Teleport Pro
psbot
Twiceler
NetinfoBot
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
grub-client
NPBot
WebReaper


* Mediapartners-Google Pinterest GPTBot ChatGPT-User CCBot ClaudeBot PerplexityBot


* GPTBot OAI-SearchBot ChatGPT-User Claude-Web PerplexityBot Perplexity-User ClaudeBot Claude-SearchBot anthropic-ai Google-Extended


* AI2Bot Amazonbot Applebot-Extended anthropic-ai Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai Diffbot DuckAssistBot FacebookBot Google-Extended GPTBot Meta-ExternalAgent Meta-ExternalFetcher OAI-SearchBot PerplexityBot PerplexityUser


*
grapeshot
ChatGPT-User
OAI-SearchBot
MistralAI-User
GPTBot
CCBot
Google-Extended
PerplexityBot
anthropic-ai
Claude-Web
ClaudeBot
cohere-ai
Applebot-Extended
MistralAI-Index
FirstRain
rogerbot
AhrefsBot
SearchmetricsBot
MJ12bot
dotbot
Nutch
trendictionbot
XoviBot
Cliqzbot
GnowitNewsbot
MaxPointCrawler
Meltawer
Digimind
Knowings
Sindup
Cision
Talkwater
TurnitinBot
ConveraCrawler
QuepasaCreep
Jetbot
NewsNow
kbcrawl
AmiSoftware
Newzbin
Ask n read
Qwam content intelligence
Zite
Youmag
Synthesio
trendybuzz
scoop.it
linkfluence
5emeRue
Augure
Corporama
readability.com
grub-client
ia_archiver
ia_archiver-web.archive.org
k2spider
libwww
wget
5erue
adequat
adequat-systems
auramundi
coexel
leadbox
mention
Moreover
mytwip
opinion-tracker
proxem
score3
trendeo
vecteurplus
verticalsearch
vsw
winello
Fetch
infoseek
MSIECrawler
Offline Explorer
sitecheck.internetseer.com
SiteSnagger
Teleport
TeleportPro
WebCopier
WebStripper
Zealbot
asknread.com
ellisphere
spotter
Riddler


* ClaudeBot ChatGPT GPTBot GPT PerplexityBot Perplexity-User claude-web Google-Extended ChatGPT-User SemrushBot Yandex


* AhrefsBot MJ12bot DotBot Scrapy Hatena Grub ia_archiver


* GPTBot OAI-SearchBot ChatGPT-User Claude-Web ClaudeBot Google-Extended PerplexityBot


GPTBot PerplexityBot ClaudeBot Google-Extended *


*
Googlebot
Googlebot-Image
Googlebot-Video
Googlebot-News
Google-InspectionTool
GoogleOther
APIs-Google
Mediapartners-Google
DuckDuckBot
Slurp
Twitterbot
Bingbot
adidxbot
MicrosoftPreview
Applebot
AdsBot-Google
AmazonAdBot
facebookexternalhit
Yahoo Ad Monitoring
parse.ly scraper
Screaming Frog SEO Spider
SemrushBot
Botify
OnCrawl
Chrome-Lighthouse
Slackbot-LinkExpanding
Slack-ImgProxy
Slackbot
Snap URL Preview Service
proximic
GumGum-Bot
TTD-Content
Pubmatic
ias_crawler
outbrain
Leikibot
weborama-fetcher
AHC/2.1
SirdataBot
peer39_crawler/1.0
EchoboxBot/1.0
CriteoBot/0.1
AdDefend Site Context Crawler/1.0
SmartologyBot
Dianomi
ChatGPT-User
GPTBot
OAI-SearchBot


* bingbot AdsBot-Google AdIdxBot Twitterbot Bytespider CCBot ChatGLM-Spider ClaudeBot PerplexityBot anthropic-ai AmazonBot meta-externalagent GPTBot Applebot-Extended


* GPTBot Google-Extended YandexBot Baiduspider Amazonbot Meta-ExternalAgent FacebookBot Applebot-extended CCBot anthropic-ai ClaudeBot Claude-web omgili cohere-ai Scrapy Diffbot ImagesiftBot TurnitinBot


* GPTBot ClaudeBot PerplexityBot Google-Extended


*
AwarioRssBot
AwarioSmartBot
Omgilibot
Omgili
proximic
peer39_crawler
peer39_crawler/1.0
Screaming Frog SEO Spider
SiteCheckerBotCrawler
AhrefsBot
sentibot
MJ12bot
BLEXBot


* GPTBot Google-Extended ClaudeBot PerplexityBot Amazonbot bingbot GnowitNewsbot ChatGPT-User OAI-SearchBot SemrushBot Applebot


*
PetalBot
Yandex
PerplexityBot
Amazonbot
AhrefsBot
archive.org_bot
ArchiveBot
Arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
BacklinkCrawler
BIXOCRAWLER
BLEXBot
Bytespider
Cliqzbot
ConveraCrawler
crawler4j
CrystalSemanticsBot
DataForSeoBot
Diffbot
discobot
discoverybot
dloader(NaverRobot)
dotbot
Download Ninja
DuckAssistBot
europarchive.org
Exabot
ExtractorPro
Fetch
Flamingo_SearchEngine
FriendlyCrawler
Gigabot
GumGum Bot
ia_archiver
ia_archiver-web.archive.org
ImagesiftBot
IstellaBot
Jetbot
JikeSpider
Jyxobot
k2spider
Kangaroo Bot
kbcrawl
LexxeBot/1.0
LinkArchiver
LinkextractorPro
Livelapbot
magpie-crawler
meta-externalfetcher
MJ12bot
Moreoverbot
MSIECrawler
nabot
NaverBot
NerdByNature.Bot
netEstate NE Crawler
nNetSeer crawler
NextGenSearchBot
Nicecrawler
NPBot
nutch
Openbot
PanguBot
peer39_crawler
peer39_crawler/1.0
PiplBot
Pixray-Seeker
Primalbot
psbot
QuerySeekerSpider
Quora-Bot
rogerbot
Scrapy
searchpreview
SemrushBot
SEOENGBot
SEOkicks-Robot
SiteBot
SiteBot/0.1
Slurp
Sosospider
spbot
Timpibot
trendictionbot
trovitBot
True_Robot
TurnitinBot
UbiCrawler
uMBot-LN
URL_Spider_Pro
verticalsearch
WBSearchBot
web-archive-net.com.bot
WeSEE:Search
Wotbot
YouBot
Zealbot
AppleBot


CCBot
CCBot/2.0
CCBot/2.0 (http://commoncrawl.org/faq/)
anthropic-ai
cohere-ai
WikiDo
fr_crawler
Yandex
Baiduspider
Baiduspider-image
Baiduspider-video
Baiduspider-favo
Baiduspider-news
Baiduspider-cpro
Baiduspider-ads
trendictionbot
Bitvorebot
BLP_bbot
heritrix
magpie-crawler
Kraken
moatbot
bhcBot
SemrushBot
Synthesio
AhrefsBot
BrandONbot
GermCrawler
Sogou
Exabot
MaxPointCrawler
ADmantX
Slurp
*
Twitterbot
AmazonAdBot


AEM Seagate
HypeStat
EtaoSpider
AppCodesCrawler
bingbot
*


trovitBot Mediapartners-Google AmazonAdBot * Googlebot


* Bingbot Googlebot ChatGPT-User OAI-SearchBot Claude-User Claude-SearchBot PerplexityBot MistralAI-User facebookexternalhit LinkedInBot Twitterbot TikTokSpider anthropic-ai ClaudeBot Bravebot GPTBot Google-Extended Applebot-Extended Meta-ExternalAgent Meta-ExternalFetcher FacebookBot Amazonbot cohere-ai cohere-training-data-crawler Diffbot AI2Bot AI2Bot-Dolma CCBot BaiChuanBot Bytespider DeepSeekBot GrokBot xAI-Grok Grok-DeepSearch TansuoBot WeChatBot Yibot ZhipuAI 360Spider Sogoubot PanguBot PetalBot YandexAdditional YandexAdditionalBot Omgilibot Omgili Timpibot VelenPublicWebCrawler Webzio-Extended ImagesiftBot img2dataset FriendlyCrawler ICC-Crawler Kangaroo+Bot SemrushBot-OCOB iaskspider/2.0 YouBot


*
Amazonbot
Applebot-Extended
Bytespider
CCBot
ClaudeBot
CloudflareBrowserRenderingCrawler
Google-Extended
GPTBot
meta-externalagent
Googlebot
AdsBot-Google
Googlebot-Image
Twitterbot
Baiduspider
008
AdsTxtCrawler
AI2Bot
Ai2Bot-Dolma
anthropic-ai
Aranet-SearchBot
Archive-It
archive.org_bot
Arquivo-web-crawler
Authory
bl.uk_lddc_bot
bne.es_bot
bnf.fr_bot
ChatGLM-Spider
Claude-Web
CloudVertexBot
cohere-ai
cohere-training-data-crawler
CohereBot
Cotoyogi
Crawl4AI
CUCH.org Archive Bot
Dataprovider.com
Datenbank Crawler
dcrawl
DeepSeekBot
FacebookBot
GoogleOther
HelloworkJobPostingBot
heritrix
HTTrack
HTTrack 3.0
ia_archiver
ia_archiver-web.archive.org
IABot
iAskBot
iaskspider
ICC-Crawler
imageSpider
IndeedJobBot
Internet Archive
Kangaroo Bot
KimiBot
KrawlerBot
KunatoCrawler
laion-huggingface-processor
LCC
MetaInspector
MetaJobBot
mirrorweb
MistralBot
netarkivindsamling
netEstate Imprint Crawler
newspaper
Nicecrawler
Nutch
nyt_scraping
Offline Explorer
omgili
OpenindexSpider
PanguBot
Potions
SBIntuitionsBot
Scrapy
ServerHunterSpider
SmarshBot
special_archiver
Spider
StatsDroneBot
Thinkbot
Timpibot
Turnitin
VelenPublicWebCrawler
webzio-extended
WRTNBot
XY-Archive-Compliance
XY-Archive-Compliance-Archiver
ZenphonyBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


Mediapartners-Google YisouSpider Leikibot ShopWiki NPBot psbot HenryTheMiragoRobot gigabot appie Meta-ExternalAgent AhrefsBot *


* GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-Web PerplexityBot Google-Extended Applebot-Extended


* GPTBot ClaudeBot anthropic-ai Google-Extended CCBot Bytespider Applebot-Extended Meta-ExternalAgent cohere-ai ImagesiftBot Omgilibot AhrefsBot SemrushBot MJ12bot DotBot PetalBot BLEXBot DataForSeoBot SerpstatBot ZoominfoBot Barkrowler magpie-crawler MauiBot SeekportBot Nimbostratus-Bot


Googlebot
Bingbot
Slurp
DuckDuckBot
Baiduspider
YandexBot
Applebot
OAI-SearchBot
ChatGPT-User
PerplexityBot
Claude-User
Claude-SearchBot
claude-code
AdsBot-Google
AdsBot-Google-Mobile
Mediapartners-Google
facebookexternalhit
Twitterbot
LinkedInBot
Pinterestbot
Slackbot
Slackbot-LinkExpanding
Discordbot
TelegramBot
WhatsApp
ias_crawler
ias_wombles
ias_admantx
DoubleVerifyBot
DVBot
proximic
peer39_crawler
GrapeshotCrawler
grapeshot
GumGum
GlancelotBot
TTD-Content
PubMatic Crawler Bot
SinceraSyntheticUser
AmazonAdBot
Leikibot
SlickBot
GPTBot
ClaudeBot
anthropic-ai
Claude-Web
Google-Extended
Google-CloudVertexBot
Meta-ExternalAgent
Meta-ExternalFetcher
FacebookBot
Applebot-Extended
Amazonbot
Bytespider
CCBot
cohere-ai
Diffbot
YouBot
DuckAssistBot
Timpibot
Omgilibot
Omgili
ImagesiftBot
GrokBot
AI2Bot
FirecrawlAgent
Scrapy
*


Mediapartners-Google * YandexBot msnbot GPTBot ChatGPT-User OAI-SearchBot Google-Extended ClaudeBot CCBot PerplexityBot DeepSeekBot bingbot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* GPTBot ChatGPT-User CCBot Bytespider Google-Extended OAI-SearchBot ClaudeBot PerplexityBot anthropic-ai PetalBot


* Google-Extended GoogleOther GoogleOther-Image GoogleOther-Video GPTBot ChatGPT-User OAI-SearchBot CCBot anthropic-ai omgili omgilibot FacebookBot Amazonbot PerplexityBot YouBot Applebot-Extended Bytespider ClaudeBot Diffbot Meta-ExternalAgent Claude-Web cohere-ai Ai2Bot Ai2Bot-Dolma FriendlyCrawler ICC-Crawler ImagesiftBot PetalBot Scrapy Timpibot VelenPublicWebCrawler Webzio-Extended DeepSeekBot


facebookexternalhit Bytespider Baiduspider PetalBot TikTokSpider Twitterbot Bingbot *


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


Googlebot Yandex GPTBot ClaudeBot PerplexityBot CCBot Google-Extended Bingbot Applebot Amazonbot *


*
"008"
anthropic-ai
Applebot
Applebot-extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
Diffbot
FacebookBot
GPTBot
HTTrack
Nutch
Offline Explorer
omgili
Scrapy
YouBot


* GPTBot Google-Extended Applebot-Extended Applebot AmazonAdBot anthropic-ai Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web Diffbot FacebookBot ImagesiftBot Omgilibot Omgili PerplexityBot YouBot Googlebot-News


*
Googlebot
YandexBot
bingbot
OAI-SearchBot
ChatGPT-User
GPTBot
ia_archiver
NaverBot
Yeti
Googlebot-Image
BaiduSpider
Sosospider
HaosouSpider
Sogou web spider
Sogou inst spider
Sogou spider
Sogou wap spider
almaden
ASPSeek
Axmo
BecomeBot
CherryPicker
Crescent Internet ToolPak
DISCo Pump
Download Ninja
DTS Agent
EmailCollector
EmailSiphon
EmailWolf
Expired Domain Sleuth
Franklin Locator
Gaisbot
grub
htdig
HTTrack
iaea.org
IconSurf
Iltrovatore-Setaccio
Indy Library
InternetSeer.com
IUPUI
larbin
libwww
LNSpiderGuy
lwp-trivial
MetaTagRobot
Missigua Locator
mozDex
MSIECrawler
NextGenSearch
NPbot
Nutch
Offline Explorer
Oracle Ultra Search
PictureOfInternet
PlantyNet
psbot
QuepasaCreep
RPT-HTTPClient
sitecheck.internetseer.com
SiteSnagger
spider.acont.de
Sqworm
SSM Agent
szukacz
TAMU
Teleport
TeleportPro
Telesoft
TurnitinBot
TutorGig
Ultraseek
WebCopier
WebEMailExtractor
WebReaper
Webster Pro
WebStripper
WebZIP
Wotbox
Xenu
Zao
Zealbot
ZipppBot
ZyBORG
AdsBot-Google


* Googlebot-Image Mediapartners-Google Adsbot-Google Twitterbot


BLEXBot Bytespider Exabot Scanbot Semrush SemrushBot YoudaoBot Baiduspider YandexBot openai Petalbot Bolt BunnySlippers ChatGPT-User GPTBot Gigabot HybridBot JikeSpider SMTBot ScreenerBot SiteLockSpider SuperBot SuperHTTP Amazonbot AhrefsBot Applebot SEMrush DataForSeoBot ClaudeBot DotBot phxbot AwarioBot SeekportBot MJ12bot Majestic12 facebookexternalhit/1.1 PerplexityBot Qwantbot CCBot python-requests/2.32.3 wget curl *


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
"008"
AddSearchBot
AI2Bot
Ai2Bot-DeepResearchEval
Ai2Bot-Dolma
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-extended
atlassian-bot
AutoRAG
AwarioSmartBot
AzureAI-SearchBot
Big Sur AI
bigsur.ai
Brandwatch
Bravebot
Brightbot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AI-Search
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
Direqt Anomura
DuckAssistBot
Echobot Bot
ExaBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
GPTBot
GROK-2
HTTrack
iAsk
iAskBot
iaskspider
ICC Crawler
ICC-Crawler
ImageSiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
Kunato
laion-huggingface-processor
LCC
LINER Bot
LinerBot
LinkupBot
Manus-User
Meta-ExternalAgent
Meta-ExternalFetcher
meta-webindexer
mistral.ai
MistralAI-User
netEstate Imprint Crawler
NovaAct
Novellum AI Crawl
Nutch
OAI-SearchBot
Offline Explorer
omgili
opencode
PanguBot
PeopleInc-DCipher-Scraper/1.1.0
Perplexity-User
PerplexityBot
PetalBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
Scrapy
SeekrBot
SemrushBot-OCOB
SemrushBotSwa
ShapBot
Spider
TavilyBot
TerraCotta
Timpibot
Trae
TwinAgent


* GPTBot ChatGPT-User PerplexityBot ClaudeBot Google-Extended Applebot-Extended


Googlebot bingbot Baiduspider DuckDuckBot YandexBot Yeti Applebot *


AmazonAdBot Google-InspectionTool ChatGPT-User GPTBot OAI-SearchBot *


01h4x.com
360Spider
404enemy
80legs
ADmantX
AIBOT
ALittle Client
ASPSeek
Abonti
Aboundex
Aboundexbot
Acunetix
AfD-Verbotsverfahren
AhrefsBot
AhrefsSiteAudit
AiHitBot
Aipbot
Alexibot
AllSubmitter
Alligator
AlphaBot
Anarchie
Anarchy
Anarchy99
Ankit
Anthill
Apexoo
Aspiegel
Asterias
Atomseobot
Attach
AwarioRssBot
AwarioSmartBot
BBBike
BDCbot
BDFetch
BLEXBot
BackDoorBot
BackStreet
BackWeb
Backlink-Ceck
BacklinkCrawler
Badass
Bandit
Barkrowler
BatchFTP
Battleztar Bazinga
BetaBot
Bigfoot
Bitacle
Black Hole
BlackWidow
Blackboard
Blow
BlowFish
Boardreader
Bolt
BotALot
Brandprotect
Brandwatch
Buck
Buddy
BuiltBotTough
BuiltWith
Bullseye
BunnySlippers
BuzzSumo
CATExplorador
CCBot
CODE87
CSHttp
Calculon
CazoodleBot
Cegbfeieh
CensysInspect
CheTeam
CheeseBot
CherryPicker
ChinaClaw
Chlooe
Citoid
Claritybot
Cliqzbot
Cloud mapping
Cocolyzebot
Cogentbot
Collector
Copier
CopyRightCheck
Copyscape
Cosmos
Craftbot
Crawling at Home Project
CrazyWebCrawler
Crescent
CrunchBot
Curious
Custo
CyotekWebCopy
DBLBot


* FacebookExternalHit Twitterbot Pinterestbot LinkedInBot TikTokSpider Storebot-Google Google-InspectionTool AmazonAdBot Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended Amazonbot meta-webindexer meta-externalagent meta-externalfetcher ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User cohere-training-data-crawler cohere-ai CCBot PanguBot PetalBot Bytespider Diffbot DuckAssistBot MistralAI-User Timpibot Webzio Webzio-Extended archive.org_bot ia_archiver ia_archiver-web.archive.org heritrix


* AhrefsBot Baiduspider CCBot Gigabot betaBot dotbot MJ12bot msnbot BLEXBot SeznamBot Viber/18.3.2.0


Amazonbot anthropic-ai AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web DataForSeoBot Diffbot FacebookBot Google-Extended GPTBot magpie-crawler NewsNow news-please omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot Scrapy TurnitinBot *


YandexBot Spinn3r Mail.RU_Bot Baiduspider Slurp AhrefsBot bingbot msnbot CCBot omgilibot omgili GoogleProducer Amazonbot applebot * googlebot


* facebookexternalhit LinkedInBot Twitterbot GPTBot Google-Extended


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


Mediapartners-Google * 008 AI2Bot Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler Diffbot EtaoSpider Meta-ExternalAgent mistral.ai PerplexityBot SeekrBot Timpibot Webzio-Extended Wrtn xAI-Web-Crawler


* Nuclei WikiDo Riddler PetalBot Zoominfobot Go-http-client Node/simplecrawler CazoodleBot dotbot/1.0 Gigabot Barkrowler BLEXBot magpie-crawler


Googlebot Googlebot-image * adsbot-google Nutch AhrefsBot AhrefsSiteAudit MJ12bot Pinterest


* Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai DataForSeoBot Diffbot DuckAssistBot FacebookBot FriendlyCrawler Google-CloudVertexBot Google-Extended GPTBot omgili semrushbot semrushbot-ba semrushbot-ct semrushbot-si semrushbot-swa serpstatbot zoombot


* Mediapartners-Google Amazonbot Bytespider cohere-ai CCBot GPTBot anthropic-ai Google-CloudVertexBot Google-Extended Applebot-Extended omgilibot omgili ClaudeBot Claude-Web Deepseek-V3 DeepSeek-V3 PerplexityBot Diffbot OAI-SearchBot NewsNow news-please Meta-ExternalAgent meta-externalagent Meta-ExternalFetcher FacebookBot magpie-crawler BLP_bbot sentibot peer_39_crawler/1.0 peer39_crawler dotbot SEMrushBot Scrapy AhrefsBot DataForSeoBot


* Baiduspider Googlebot AhrefsBot meta-externalagent anthropic-ai ClaudeBot openai ChatGPT bingbot Yandex *bot


Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/w4mwnpbXf3MFAbxOkJRw; +http://www.echobox.com)

Mozilla/5.0 (compatible; NewsRoom.BI/0.1; +http://www.newsroom.bi/bot.html)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36(KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; mrfCompass-Marshall/1.0)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Jukebox/1.0)


* AdsBot-Google-Mobile AdsBot-Google PetalBot dotbot AhrefsBot


* AdsBot-Google Googlebot Googlebot-Image PetalBot


*
Baiduspider
PetalBot
yisouspider
008
ChatGPT-User
ChatGPT-User/2.0
PerplexityBot
Perplexity-User
DuckAssistBot
Applebot
bingbot
Googlebot
ClaudeBot
claude-web
Claude-SearchBot
YouBot
Applebot-Extended
MistralAI-User
meta-webindexer
GoogleAgent-Mariner
DotBot
archive.org_bot
Pinterestbot
TelegramBot
kakaotalk-scrap
getstream.io/opengraph-bot
RamblerMail
Brave
OAI-SearchBot
GPTBot
anthropic-ai
Google-Extended
cohere-ai
CCBot
AI2Bot
Amazonbot
Amazonbot-Video
Bytespider
meta-externalagent
Diffbot
omgili
TimpiBot
SeznamBot
Exabot
YandexBot
Sogou
360Spider
YisouSpider
Baiduspider-Render/2.0
MJ12bot
DataForSeoBot
AliyunSecBot
ArchiveTeam
ArchiveTeam ArchiveBot
AwarioBot
ZoominfoBot
ali-implementer
Blueno
BIGO-baiguoyuan
WorksOgCrawler
OnPageBot
DoCoMo
SAMSUNG-SGH-E250
TA-Googlebot


* Googlebot-Image zs-crawler zoho-scrapy


* Googlebot-News Amazonbot anthropic-ai Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai Diffbot FacebookBot Google-Extended GPTBot magpie-crawler omgili omgilibot PerplexityBot Google-CloudVertexBot meta-externalagent meta-externalfetcher Ahrefsbot Archive.org_bot Bravebot Claude-Searchbot Claude-User DeepSeekBot Meta-WebIndexer MistralAI-Index MistralAI-User OAI-AdsBot OAI-Searchbot Perplexity-User Youbot


* Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai Diffbot FacebookBot Google-Extended GPTBot ImagesiftBot Meta-ExternalAgent meta-externalagent NewsNow news-please OAI-SearchBot omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot Scrapy


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
Yandex
ltx71 - (http://ltx71.com/)
MJ12bot
SemrushBot
TelegramBot


Baiduspider
Googlebot
sogou spider
Sosospider
YodaoBot
Bingbot
Slurp
MSNBot
360Spider
YisouSpider
Chinasospider
*


Mediapartners-Google
Google-Site-Verification
Adsbot-Google
GoogleOther
AmazonAdBot
*
bingbot
GPTBot
PerplexityBot
Bytespider
Amazonbot
ClaudeBot
CCBot
TweetmemeBot
QuerySeekerSpider
NING
SputnikBot
Cliqzbot
Flamingo_SearchEngine
Vegi bot
demandbase-bot
trendkite-akashic-crawler
DemandbasePublisherAnalyzer
GrapeshotCrawler
YaK
MauiBot
GermCrawler
PaperLiBot
BLEXBot
PetalBot
DataForSeoBot
Buck
SeekportBot


GPTBot ChatGPT-User OAI-SearchBot ClaudeBot PerplexityBot Perplexity-User Google-Extended Applebot-Extended


GPTBot
ChatGPT-user
Google-Extended
Google-NotebookLM
Gemini-Deep-Research
OAI-SearchBot
CCbot
anthropic-ai
Claude-Web
ClaudeBot
Claude-User
Claude-SearchBot
omgili
omgilibot
PerplexityBot
Perplexity-User
DuckAssistBot
DeepSeekBot
DeepSeek
Amazonbot
Amazonbot-Video
Applebot-Extended
Applebot
Bytespider
cohere-ai
Timpibot
YouBot
Brightbot 1.0
FirecrawlAgent
PetalBot
MistralAI-User
VelenPublicWebCrawler
wpbot
iAskBot
PanguBot
meta-externalagent
Ai2Bot-Dolma
ZanistaBot
*


* Bingbot facebookexternalhit Pinterestbot Twitterbot


anthropic-ai ClaudeBot Claude-Web GPTBot ChatGPT-User OAI-SearchBot CCBot PerplexityBot Google-Extended Applebot-Extended Bytespider Amazonbot Meta-ExternalAgent cohere-ai *


* Applebot ImagesiftBot OAI-SearchBot GPTBot EzoicBot PerplexityBot ClaudeBot meta-externalagent meta-externalfetcher Amazonbot ShapBot Googlebot Mediapartners-Google ia_archiver Twitterbot Yandex DuckDuckBot bingbot


* AmazonAdBot bytespider mistralai cohere ai2bot youbot omgili diffbot kangaroo img2dataset amazonbot amazon-qbusiness meta-externalagent meta-webindexer


* GPTBot ClaudeBot PerplexityBot Google-Extended


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


facebookexternalhit meta-externalagent * Google-Extended Gemini-Deep-Research ClaudeBot anthropic-ai AppleBot-Extended


* GPTBot Amazonbot CCBot anthropic-ai Claude-Web Google-Extended


JoobleBot PetalBot * Mediapartners-Google AdIdxBot Google-Display-Ads-Bot


* Mediapartners-Google bingbot CCBot Bytespider


*
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
BingPreview
Baiduspider
CCBot
ChatGPT-User
Claude-Web
Claude-User
ClaudeBot
Claude-SearchBot
cohere-ai
cohere-training-data-crawler
Diffbot
DuckAssistBot
DeepSeekBot
FacebookBot
FriendlyCrawler
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
Gemini-Deep-Research
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
LinkedInBot
Meta-ExternalAgent
Meta-ExternalFetcher
MistralAI-User
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
Perplexity-User
PetalBot
Scrapy
SemrushBot
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot
YandexBot
facebookexternalhit
Twitterbot
Slackbot-LinkExpanding
Googlebot
Googlebot-News
Googlebot-Image
Googlebot-Mobile
Storebot-Google
AdsBot-Google
AdsBot-Google-Mobile
Bingbot
Yandex
Google-InspectionTool
Google-Site-Verification
Slurp
DuckDuckBot
Naverbot
Bingbot/Render
AdIdxBot
MSNBot-Media
AccessibleWebBot


* AmazonAdBot Hootsuite HootsuiteBot OgScrper OgScraper Hootsuite-Authoring


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


Googlebot
Mediapartners-Google
Yahoo! Slurp
bingbot
Yandex
Baiduspider
Sogou
ia_archiver
IPS-Agent
BLEXBot
Pandalytics
IonCrawl
*


trendkite-akashic-crawler Bytedance Bytespider RSiteAuditor XoviOnpageCrawler BLEXBot RuxitSynthetic 8LEGS *


* grapeshot GPTBot ClaudeBot Google-Extended Amazonbot CCBot


*
Google-CloudVertexBot
Google-Extended
ChatGPT-User
GPTBot
OAI-SearchBot
OAISearch
anthropic-ai
claude-web
claudebot
facebookbot
meta-externalagent
meta-externalfetcher
applebot-extended
perplexitybot
cohere-ai
youbot
amazonbot
aliyunsecbot
audigentadbot
awariorssbot
awariosmartbot
blexbot
bytespider
ccbot
dataforseobot
diffbot
duckassistbot
echoboxbot
friendlycrawler
ia_archiver
imagesiftbot
mycentralaiscraperbot
newsnow
news-please
omgili
omgilibot
webzio-extended
peer39_crawler
peer39_crawler/1.0
quillbot.com
quora-bot
scrapy
seekrbot
seznamhomepagecrawler
taragroup intelligent bot
timpibot
turnitinbot
viennatinybot
jetslide
magpie-crawler
poseidon research crawler


* GPTBot PerplexityBot bingbot BingPreview bingGPT Googlebot Google-Extended Bard Gemini ClaudeBot Slurp


* GPTBot ChatGPT-User Google-Extended PerplexityBot Claude-Web Bingbot Meta-ExternalAgent DeepSeekBot Dalvik *Dalvik*


* Googlebot-News Amazonbot anthropic-ai Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai Diffbot FacebookBot Google-Extended GPTBot magpie-crawler omgili omgilibot PerplexityBot meta-externalagent meta-externalfetcher Google-CloudVertexBot Ahrefsbot Archive.org_bot Bravebot Claude-Searchbot Claude-User DeepSeekBot Meta-WebIndexer MistralAI-Index MistralAI-User OAI-AdsBot OAI-Searchbot Perplexity-User Youbot


AmazonAdBot Google-InspectionTool ChatGPT-User GPTBot OAI-SearchBot *


Googlebot Googlebot-image Storebot-Google Google-Extended *


* GPTBot ChatGPT-User Claude-Web ClaudeBot PerplexityBot Applebot-Extended cohere-ai Bytespider CCBot MistralAI-User Google-Extended


* Googlebot Yeti Naverbot ia_archiver AhrefsBot


MJ12bot
Mediapartners-Google*
IsraBot
Orthogaffe
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
fast
wget
grub-client
k2spider
NPBot
WebReaper
SemrushBot
*


*
GPTBot
PerplexityBot
Amazonbot
anthropic-ai
AwarioRssBot
AwarioSmartBot
Bytespider
CCBot
HuggingFaceBot
ChatGPT-User
ClaudeBot
Claude-Web
DataForSeoBot
Diffbot
FacebookBot
Google-Extended
Google-NotebookLM
Google-Agent
magpie-crawler
NewsNow
news-please
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
Scrapy
TurnitinBot
cohere-ai
Applebot-Extended
Meta-ExternalAgent
meta-externalagent
OAI-SearchBot
Claude-User
polymarket-news-bot
KalshiBot
polymarket-bot
OpenClaw
gold-ai-event-bridge
TradingBot
trading-pipeline
kalshi-bot
BivariateKalshiPipeline
Falcon Trading Platform
StockMarketAnalyser
FinanceApp
polyclaw-ingester
IsoxyTrader
Neo Trading Bot
MerlinChartWizard
investment-dashboard


* GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-Web anthropic-ai Google-Extended PerplexityBot Applebot-Extended CCBot bingbot NerdyBot duggmirror Bytespider


ai2bot-dolma
anthropic-ai
applebot-extended
bytespider
ccbot
claude-user
claude-web
claudebot
chatGPT-User
cohere-ai
diffbot
duckassistbot
Friendlycrawler
GPTBot
Grokbot
xAI-Grok
xAI
omgili
omgilibot
perplexity-user
perplexitybot
scrapy
timpibot
youbot
webzio-extended
kangaroo-bot
Spinn3r
TaraGroup Intelligent Bot
google-extended
AmazonAdBot
*


Mozilla/4.0 (compatible; BullsEye; Windows 95)

Crescent Internet ToolPak HTTP OLE Control v.1.0


* Yandex Baiduspider GPTBot Meta-ExternalAgent


Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot Cohere-ai GPTBot Google-CloudVertexBot Google-Extended Img2dataset PanguBot PetalBot SentiBot YouBot magpie-crawler meta-externalagent * ChatGPT-User ChatGPT-User/2.0 Gemini-Deep-Research OAI-SearchBot applebot PerplexityBot bingbot Yandex


archive.org_bot bingbot DuckDuckBot Googlebot Googlebot-Mobile ia_archiver Yandex Adsbot-Google AhrefsBot Googlebot-Image Mail.Ru Mediapartners-Google msnbot msnbot-media Slurp StackRambler Yahoo-MMCrawler *


* ClaudeBot Amazonbot facebookexternalhit


PetalBot
bedrock-agentcore-browser
Amazon-Bot
Amazonbot
AmazonProductDiscoveryBot
Amzn-SearchBot
Amzn-User
Arquivo-web-crawler
bnf.fr_bot
Bytespider
Claude-Extension
ClaudeBot
Claude-User
Claude-SearchBot
Comet
CloudFlare-AlwaysOnline
Cloudflare-Healthchecks
CloudflareBrowserRendering
Cloudflare-Radar
CCBot
DeepSeekBot
DuckAssistBot
archive.org_bot
ia_archiver
MistralAI-User
Moreover
Adbeat
Argus
Baiduspider
Baiduspider-image
Bitdefender
Blogtrottr
BloombergBot
BoardReader
checkmy.ws
Clickagy
CookieBot
cXensebot
DataObserver
DomainStatsBot
Embedly
Feedspotbot
FlipboardProxy
Getintent
Google-Apps-Script
GoogleDrive
Grapevine6
Gumgum
HaloBot
HubSpot Crawler
IbouBot
Inout
LinerBot
MJ12bot
Manus
Meltwater
Metajob
Muckrack
NetSeer
Netvibes
Onyphe
OpenClaw
Opoint
OutbrainBot
PlumAnalytics
ProSitemaps
PerplexityBot
Perplexity-User
proximic
PubMatic
Quantcastbot
SearchConcepts
Semantic-Visions
SentiBot
Sindup
Sirdata
SmartologyBot
Superfeedr
surlybot
Synaptive
TinEye
Trendiction
TurnitinBot
Ubermetrics
Vebidoo
Webcorp
Weborama
WebPageTest
YandexAccessibilityBot
YandexDirect
YandexDirectDyn
YaDirectFetcher
YandexFavicons
YandexMetrika
YandexMobileBot
YandexNews
YandexRCA
YandexRenderResourcesBot
YandexUserproxy
*


* ADmantX AlphaBot anthropic-ai AwarioRssBot AwarioSmartBot BLEXBot Buzzbot Bytespider CCBot ChatGPT-User claritybot Claude-Web ClaudeBot cohere-ai Diffbot FacebookBot FriendlyCrawler Google-Extended GPTBot huggingface ImagesiftBot img2dataset magpie-crawler Meltwater Neevabot news-please NewsNow Nutch omgili omgilibot panscient.com Perplexity-ai PerplexityBot PetalBot PiplBot scoop.it Scrapy Seekr SentiBot SeznamBot TurnitinBot YouBot ZumBot


* Googlebot Mediapartners-Google Covario-IDS AhrefsBot Owlin WBSearchBot PiplBot XoviBot magpie-crawler Yandex


Slurp Applebot-Extended SeekportBot AhrefsBot StractBot AwarioRssBot AwarioSmartBot ias_crawler MJ12bot Cliqzbot DotBot/1.1 MauiBot NTENTbot CCBot sentibot trendictionbot yacybot iSec_Bot BLEXBot AhrefsBot/6.1 Qwantify/Bleriot/1.1 DemandbasePublisherAnalyzer/0.1 weborama-fetcher ClaudeBot * AmazonAdBot


* GPTBot ClaudeBot Claude-User Claude-SearchBot Google-Extended CCBot FacebookBot Omgilibot


* AhrefsBot Amazonbot anthropic-ai Applebot Applebot-Extended archive.org_bot Archive-It BaiduSpider BLP_bbot Brandwatch Bytespider CCBot ChatGPT-User Cision Claude ClaudeBot Claude-SearchBot Claude-Web DataForSeoBot Diffbot dotbot/1.0 FacebookBot Gigabot Gnowit GnowitNewsbot Google-Extended GPTBot Heritrix ia_archiver linkfluence magpie-crawler mediatoolkit.com Meltwater Meta-ExternalAgent Meta-ExternalFetcher Muckrack OAI-SearchBot openai OTMedia Perplexity PerplexityBot scoopit scpitspi-rs semantic-visions.com SemrushBot SentiBot trendiction Turnitin webzio webzio-extended YouBot


* Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot Claude-Web ClaudeWeb cohere-ai Diffbot FacebookBot FriendlyCrawler Google-Extended GPTBot ImagesiftBot img2dataset Meta-ExternalAgent omgili omgilibot Scrapy Timpibot VelenPublicWebCrawler YouBot


* Mediapartners-Google Adsbot-Google Googlebot-Image Googlebot-Mobile ClaudeBot


Mediapartners-Google*
NetShelter ContentScan
NetShelter
^NetShelter
sentibot
anthropic-ai
Applebot-Extended
AwarioRssBot
AwarioSmartBot
Bytespider
CCBot
ClaudeBot
Claude-Web
cohere-ai
Diffbot
FacebookBot
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
GPTBot
ImagesiftBot
magpie-crawler
Meta-ExternalAgent
meta-externalagent
MyCentralAIScraperBot
NewsNow
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Poseidon Research Crawler
quillbot.com
Scrapy
Timpibot
*


* msnbot Bingbot AdIdxBot BingPreview Yandex GPTBot ChatGPT-User CCBot SemrushBot


*
Googlebot
Googlebot-News
Googlebot-Image
Googlebot-Video
Googlebot-Mobile
AdsBot-Google
AdsBot-Google-Mobile
Mediapartners-Google
Google-Extended
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
UbiCrawler
DOC
Zao
Slurp
newsproxy.app
PerplexityBot
TurnitinBot
magpie-crawler
omgili
omgilibot
DataForSeoBot
Diffbot
news-please
AwarioRssBot
AwarioSmartBot
Bytespider
archive.org_bot
archive-it.org
Maxthon
CNCDialer
CCBot
ChatGPT Agent
ClaudeBot
Claude-Web
GPTBot
anthropic-ai
ChatGPT-User
OAI-SearchBot
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Meta-WebIndexer


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


* YandexAccessibilityBot YandexCalendar YandexDialogs YandexDirect YandexDirectDyn YandexFavicons YaDirectFetcher YandexMobileBot YandexMarket YandexMetrika YandexMobileScreenShotBot YandexOntoDBAPI YandexPartner YandexRCA YandexRenderResourcesBot YandexSearchShop YandexTracker YandexUserproxy YandexVideoParser YandexScreenshotBot YandexAdditionalBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
AmazonAdBot
AI2Bot
Bytespider
CCBot
Cohere-ai
Diffbot
FirecrawlAgent
ImagesiftBot
Kangaroo Bot
Meta-ExternalAgent
Omgilibot
PanguBot
Timpibot
Webzio-Extended
HTTrack
Nutch
Scrapy


* Flamingo_SearchEngine ExaBot dotbot genieo crystalsemantics grapeshot ClaudeBot AmazonAdBot


*
"008"
anthropic-ai
Applebot
Applebot-extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
Diffbot
FacebookBot
GPTBot
HTTrack
Nutch
Offline Explorer
omgili
Scrapy
YouBot


* OAI-SearchBot ChatGPT-User PerplexityBot Claude-SearchBot Claude-User YouBot DuckAssistBot GPTBot Google-Extended ClaudeBot Amazonbot CCBot Bytespider Meta-ExternalAgent Applebot-Extended cohere-ai Diffbot Baiduspider


* bingbot Baiduspider YandexBot Slurp GPTBot ChatGPT-User ChatGPT-User/2.0 OAI-SearchBot CCBot anthropic-ai ClaudeBot claude-web PerplexityBot Perplexity-User MistralAI-User Google-Extended Amazonbot Applebot-Extended cohere-ai Diffbot omgili YouBot Bytespider AhrefsBot SemrushBot MJ12bot DotBot PetalBot BLEXBot DataForSeoBot HTTrack wget curl WebCopier WebZIP Teleport WebReaper WebSauger WebStripper WebSucker WebWhacker emailcollector emailsiphon emailwolf Nmap Nikto sqlmap masscan ZmEu Acunetix Nessus OpenVAS


Amazonbot PerplexityBot Perplexity-User ClaudeBot Claude-User GPTBot ChatGPT-User FacebookExternalHit FacebookBot Twitterbot LinkedInBot Bingbot *


* bingbot AdsBot-Google AdIdxBot Twitterbot Bytespider CCBot ChatGLM-Spider ClaudeBot PerplexityBot anthropic-ai AmazonBot meta-externalagent GPTBot Applebot-Extended


Semrushbot-SA Semrushbot-SI PageFreezer *


* OAI-SearchBot ChatGPT-User Claude-SearchBot Claude-User PerplexityBot Perplexity-User GPTBot anthropic-ai ClaudeBot Claude-Web CCBot Bytespider Meta-ExternalAgent Meta-ExternalFetcher cohere-ai cohere-training-data-crawler Google-Extended AzureAI-SearchBot bedrockbot Amazonbot Diffbot img2dataset Applebot-Extended


* AmazonAdBot Applebot-Extended Bytespider CCBot ClaudeBot Google-Extended GPTBot meta-externalagent


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* GPTBot OAI-SearchBot ChatGPT-User Google-NotebookLM Google-Extended bingbot BingPreview MicrosoftPreview ClaudeBot Claude-SearchBot PerplexityBot Perplexity-User CCBot CazoodleBot MJ12bot dotbot/1.0 Gigabot


Yandex Googlebot * SemrushBot BLEXBot AhrefsBot MJ12bot OAI-SearchBot ChatGPT-User ChatGPT-User/2.0 GPTBot anthropic-ai ClaudeBot claude-web PerplexityBot Perplexity-User Google-Extended BingBot Amazonbot Applebot Applebot-Extended FacebookBot meta-externalagent LinkedInBot DuckAssistBot Bytespider cohere-ai AI2Bot CCBot Diffbot omgili TimpiBot YouBot


*
Baiduspider
PetalBot
yisouspider
008
ChatGPT-User
ChatGPT-User/2.0
PerplexityBot
Perplexity-User
DuckAssistBot
Applebot
bingbot
Googlebot
ClaudeBot
claude-web
Claude-SearchBot
YouBot
Applebot-Extended
MistralAI-User
meta-webindexer
GoogleAgent-Mariner
DotBot
archive.org_bot
Pinterestbot
TelegramBot
kakaotalk-scrap
getstream.io/opengraph-bot
RamblerMail
Brave
OAI-SearchBot
GPTBot
anthropic-ai
Google-Extended
cohere-ai
CCBot
AI2Bot
Amazonbot
Amazonbot-Video
Bytespider
meta-externalagent
Diffbot
omgili
TimpiBot
SeznamBot
Exabot
YandexBot
Sogou
360Spider
YisouSpider
Baiduspider-Render/2.0
MJ12bot
DataForSeoBot
AliyunSecBot
ArchiveTeam
ArchiveTeam ArchiveBot
AwarioBot
ZoominfoBot
ali-implementer
Blueno
BIGO-baiguoyuan
WorksOgCrawler
OnPageBot
DoCoMo
SAMSUNG-SGH-E250
TA-Googlebot


Mediapartners-Google SeekportBot Yandex GPTBot GPTCrawler ChatGPT-User OAI-SearchBot ClaudeBot anthropic-ai Google-Extended CCBot PerplexityBot Perplexity-User Bytespider Bytespider-webpage Amazonbot *


Twitterbot facebot facebookexternalhit bingbot *


Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
Meta-ExternalAgent
Meta-ExternalFetcher
FacebookBot
facebookexternalhit
*
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
Diffbot
DuckAssistBot
FriendlyCrawler
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
OAI-SearchBot
omgili
omgilibot
PerplexityBot
PetalBot
Scrapy
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot


Yandex GoogleBot * msnbot bingbot dotbot SemrushBot weborama-fetcher AhrefsBot DataForSeoBot SeekportBot serpstatbot BLEXBot ImagesiftBot ClaudeBot


*
Amazonbot
AlibabaBot
Ai2Bot-Dolma
Applebot-Extended
Bytespider
CCBot
ChatGLM-Spider
ClaudeBot
Claude-Web
CloudVertexBot
cohere-training-data-crawler
Cotoyogi
Datenbank Crawler
Diffbot
FacebookBot
Google-Extended
GoogleOther
GPTBot
ICC-Crawler
imageSpider
Kangaroo Bot
laion-huggingface-processor
LCC
meta-externalagent
netEstate Imprint Crawler
omgili
PanguBot
PerplexityBot
SBIntuitionsBot
Spider
Timpibot
VelenPublicWebCrawler
webzio-extended
YouBot
008
Dataprovider.com
dcrawl
HelloworkJobPostingBot
HTTrack
HTTrack 3.0
KrawlerBot
MetaInspector
MetaJobBot
newspaper
Nutch
Offline Explorer
OpenindexSpider
Potions
Scrapy
ServerHunterSpider
StatsDroneBot
CloudflareBrowserRenderingCrawler
aa
AhrefsBot
Attracta
Barkrowler
BrightEdge Crawler
BLEXBot
CaliberBot
Caliperbot
ClarityBot
Clearscopebot
Cloudtrellis
cludo.com
Cocolyzebot
cognitiveSEO Crawler
contentking
Convermax
Cxense
DataForSeoBot
DataForSEO Bot
DomainStatsBot
DotBot
Dragonbot
Huckabot
Huckabuy Bot
hypestat
iaskspider
img2dataset
James BOT
LinkCheck by Siteimprove
LinkChecker Bot
linkchecker.pro
linkdexbot
LinksIndexerBot
MarketGoo
MBCrawler
MegaIndex.ru
MJ12bot
Moz dotbot
Moz rogerbot
Nitro-
NitroBot
online-webceo-bot
Prerender
Pro Sitemaps
Readable
RevvimGort
rogerbot-crawler


* GoogleBot Yandex ClaudeBot Bytespider CCBot FacebookBot GPTBot


* MJ12bot Twitterbot GPTBot Google-Extended trendkite-akashic-crawler


*
Googlebot
Facebot
Twitterbot
ChatGPT-User
AdsBot-Google
AdsBot-Google-Mobile
Google-HotelAdsVerifier
Google-HotelAdsVerifier/2.0
Google-Travel-Flights
Mediapartners-Google
AdIdxBot
BUbiNG
proximic
A6-Indexer/1.0
ADmantX Platform Semantic Analyzer
ContextAd Bot
berlin-fu-cow
ias-ir
ias-sg
AhrefsBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* MJ12bot AhrefsBot SemrushBot dotbot CCBot Amazonbot CriteoBot/0.1


*
DotBot
dotbot
Sogou Spider
Exabot
YoudaoBot
MegaIndex.ru
megaindex.com
SeznamBot
Cliqzbot
Gigabot
spbot
Ezooms
ScoutJet
BLEXBot
sistrix
SISTRIX Crawler
SISTRIX
SEOkicks
SEOkicks-Robot
SearchmetricsBot
SurveyBot
CCBot
DomainStatsBot
Sosospider
Teleport
TeleportPro
HTTrack


*
Googlebot
Googlebot-News
Googlebot-Image
Googlebot-Video
AmazonAdBot
Baiduspider
Yandex
008
UnisterBot
JikeSpider
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
larbin
libwww
ZyBORG
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Download Ninja
Maxthon
CNCDialer
wget
grub-client
k2spider
NPBot
WebReaper
TurnitinBot
MJ12bot
magpie-crawler
Flamingo_SearchEngine
Exabot
Moreoverbot
CyberAlert
Newscan
Spinn3r
LexxeBot/1.0
NextGenSearchBot
Sosospider
rogerbot
SiteBot/0.1
SiteBot
CrystalSemanticsBot
NetSeer crawler
trovitBot
DotBot
Ezooms
discobot
Jyxobot
sogou
heritrix
NerdByNature.Bot
psbot
WBSearchBot
http://addthis.com/
discoverybot
bl.uk_lddc_bot
IstellaBot
seokicks
Bender
wotbox
Yasni
netEstate NE Crawler
Pixray-Seeker
integromedb
BLEXBot
BDCbot
WeSEE:Search
admantx
spbot
BUbiNG
anthropic-ai
AI2Bot
Applebot-Extended
Bytespider
CCBot
ClaudeBot
cohere-training-data-crawler
Cotoyogi
Datenbank Crawler
Diffbot
FacebookBot
Factset_spyderbot
GPTBot
ICC-Crawler
Kangaroo Bot
Meta-ExternalAgent
netEstate Imprint Crawler
omgili


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


* Amazonbot SlySearch SmartDownload SpankBot spanner SuperBot SupperHTTP Surfbot ia_archiver


* FacebookExternalHit Twitterbot Pinterestbot LinkedInBot TikTokSpider Storebot-Google Google-InspectionTool AmazonAdBot Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended Amazonbot meta-webindexer meta-externalagent meta-externalfetcher ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User cohere-training-data-crawler cohere-ai CCBot PanguBot PetalBot Bytespider Diffbot DuckAssistBot MistralAI-User Timpibot Webzio Webzio-Extended archive.org_bot ia_archiver ia_archiver-web.archive.org heritrix


AmazonAdBot * seekbot Exabot Slurp Cliqzbot Arquivo-web-crawler AI2Bot ChatGLM ChatGLM-spider cohere-training-data-crawler Diffbot OmigiliBot PetalBot YouBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


CCBot ChatGPT-User GPTBot Google-Extended Google-CloudVertexBot Omgilibot Omgili FacebookBot ClaudeBot Claude-SearchBot Claude-User Diffbot DuckAssistBot PerplexityBot Perplexity-User cohere-ai cohere-training-data-crawler Meta-ExternalAgent Meta-ExternalFetcher Meta-WebIndexer Timpibot Webzio-Extended YouBot Amazonbot Bytespider anthropic-ai OAI-SearchBot VelenPublicWebCrawler Gemini-Deep-Research ICC-Crawler MistralAI-User *


Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot Cohere-ai GPTBot Google-CloudVertexBot Google-Extended Img2dataset PanguBot PetalBot SentiBot YouBot magpie-crawler meta-externalagent * ChatGPT-User ChatGPT-User/2.0 Gemini-Deep-Research OAI-SearchBot applebot PerplexityBot bingbot Yandex


LinkedInBot
Scope3/2.0 (scope3.com)
*
Twitterbot
facebookexternalhit
OAI-SearchBot
PerplexityBot
ClaudeBot
Google-Extended
GPTBot
anthropic-ai
omgilibot
omgili
Mediatoolkitbot
PolecatBot
DotBot


Googlebot, bingbot, BingPreview, Yandex, DuckDuckBot, AhrefsBot


* GPTBot ChatGPT-User Google-Extended ClaudeBot PerplexityBot


MJ12bot Googlebot-News Googlebot Bingbot Slurp DuckDuckBot Baiduspider YandexBot *


* ia_archiver archive.org_bot heritrix My-heritrix-crawler Timpibot omgili Bytespider DuckAssistBot Applebot-Extended Google-Extended Google-CloudVertexBot GoogleOther GPTBot CCBot Amazonbot FacebookBot cohere-ai cohere-training-data-crawler Omgilibot ClaudeBot Diffbot PanguBot MistralAI-User webz.io DeepSeekBot AmazonAdBot Amzn-SearchBot Amzn-User Claude-SearchBot Claude-User PetalBot Meta-ExternalFetcher Meta-WebIndexer OAI-SearchBot ChatGPT-User PerplexityBot Perplexity-User


GPTBot ChatGPT-User ClaudeBot Google-Extended Applebot-Extended anthropic-ai cohere-ai CCBot ICC-Crawler Bytespider OAI-SearchBot PerplexityBot Meta-ExternalAgent Perplexity-ai Perplexity-User


* GPTBot Google-Extended CCBot FacebookBot anthropic-ai


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* Twitterbot FacebookBot ChatGPT-User Google-Extended GPTBot PerplexityBot Amazonbot anthropic-ai AwarioRssBot AwarioSmartBot Bytespider CCBot ClaudeBot Claude-Web cohere-ai DataForSeoBot Diffbot magpie-crawler NewsNow news-please omgili omgilibot peer39_crawler peer39_crawler/1.0 Scrapy TurnitinBot Botify


* Storebot-Google Mediapartners-Google AdsBot-Google AdsBot-Google-Mobile-Apps DataForSeoBot GPTBot


dotbot Slurp twiceler ScoutJet Fasterfox *


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


*
008
AddThis.com
admantx
AhrefsBot
BDCbot
Bender
BIXOCRAWLER
bl.uk_lddc_bot
BLEXBot
BUbiNG
Cliqzbot
CNCDialer
crawler4j
CrystalSemanticsBot
CyberAlert
DigExt
discobot
discoverybot
dloader
DOC
dotbot
Download Ninja
DTS Agent
Exabot
Ezooms
FairShare
Fetch
Flamingo_SearchEngine
Genieo
Gigabot
grub-client
Heritrix
heritrix/3.3.0
HTTrack
ia_archiver
integromedb
IstellaBot
JikeSpider
Jyxobot
k2spider
Kimengi
Kimengi/nineconnections.com
larbin
LexxeBot/1.0
libwww
linko
Livelapbot
magpie-crawler
Maxthon
MetaURI
Microsoft.URL.Control
MJ12bot
Moreover
Moreoverbot
MSIECrawler
nabot
NerdByNature.Bot
netEstate NE Crawler
NetSeer crawler
Newscan
NextGenSearchBot
NPBot
nutch
Offline Explorer
omgilibot
Orthogaffe
PiplBot
Pixray-Seeker
proximic
psbot
QuerySeekerSpider
rogerbot
seokicks
SEOkicks-Robot
SiteBot
SiteBot/0.1
sitecheck.internetseer.com
SiteSnagger
sogou
Sosospider
spbot
Spinn3r
Teleport
TeleportPro
trendictionbot
trovitBot
TurnitinBot
UbiCrawler
uMBot-LN
UnisterBot
UniversalFeedParser
WBSearchBot
WebCopier
WebReaper
WebStripper
WebZIP
WeSEE:Search
wget
Wotbot


008 * ChatGPT-User OAI-SearchBot Perplexity-User PerplexityBotClaude-SearchBot Claude-User


Adsbot-Google
AdsBot-Google-Mobile
APIs-Google
Applebot
Bingbot
bnf.fr_bot
exabot
Googlebot
Googlebot-Image
Googlebot-Mobile
Googlebot-News
Googlebot-Video
Googlebot_Nauxeo
ia_archiver
Mediapartners-Google
Proximic
Slurp
Storebot-Google
Twitterbot
upday
sistrix
BeOpBot
weborama-fetcher
Flipboard
FlipboardProxy
AmazonAdBot
Meltawer
Digimind
Knowings
Sindup
Cision
Talkwater
TurnitinBot
ConveraCrawler
Jetbot
NewsNow
kbcrawl
AmiSoftware
Newzbin
Ask n read
Qwam content intelligence
Zite
Youmag
Synthesio
trendybuzz
spotter
scoop.it
linkfluence
5emeRue
Augure
Corporama
grub-client
ia_archiver-web.archive.org
k2spider
libwww
wget
5erue
adequat
adequat-systems
auramundi
coexel
ellisphere
leadbox
mention
Moreover
mytwip
opinion-tracker
proxem
score3
trendeo
vecteurplus
verticalsearch
vsw
winello
Fetch
infoseek
MSIECrawler
Offline Explorer
sitecheck.internetseer.com
Teleport
TeleportPro
WebCopier
WebStripper
Zealbot
asknread.com
omgilibot
omgili
Argus
augure
Bloodhound
ChatGPT-User
cydralspider
Download Ninja
downloadexpress
EDD
eureka
Europresse
Explore
Factiva
Fasterfox


* GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-Web PerplexityBot Google-Extended Twitterbot FacebookBot


Baiduspider 360Spider Sogouspider Yisouspider PetalBot Bytespider


*
ChangeDetection
008
Nutch
mauibot
semrushbot
ahrefsbot
blexbot
seo spider
Googlebot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* 008 Googlebot AdsBot-Google Googlebot-Image


AmazonAdBot * Googlebot-News omgilibot omgili


Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot Cohere-ai GPTBot Google-CloudVertexBot Google-Extended Img2dataset PanguBot PetalBot SentiBot YouBot magpie-crawler meta-externalagent * ChatGPT-User ChatGPT-User/2.0 Gemini-Deep-Research OAI-SearchBot applebot PerplexityBot bingbot Yandex


googlebot Google-Extended Google-News Googlebot-Image/1.0 Googlebot-Video/1.0 bingbot Bravebot Applebot Pinterestbot Slackbot LinkedInBot Snapchat facebookexternalhit/1.1 TelegramBot msnbot Slurp WhatsApp ChatGPT-User OAI-SearchBot PerplexityBot ClaudeBot Claude-SearchBot OAI-AdsBot AdsBot-Google AdsBot-Google-Mobile Mediapartners-Google adidxbot meta-externalads GPTBot Applebot-Extended Meta-ExternalAgent CCBot *


SemrushBot * FacebookBot GPTBot Amazonbot PetalBot DotBot BLEXBot Qwantbot CCBot AhrefsBot MJ12bot YandexBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
SiteimproveBot-Crawler
FAST-WebCrawler/0.2
Yanga WorldSearch Bot v1.1/beta
slurp
Twiceler-0.9
cludo


*
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper


Mediapartners-Google MJ12bot * facebookexternalhit Twitterbot


ChatGPT-User GPTBot CCBot Google-Extended *


PetalBot AspiegelBot SemrushBot DotBot MauiBot MJ12bot ClaudeBot Bytedance Amazonbot AhrefsBot googlebot dlvr.it gptbot *


okhttp SkyworkSpider YisouSpider SpaHttpPing Go-http-client ImagesiftBot ClaudeBot Ninja GPTBot *


Googlebot Googlebot-image Ultraseek search.ch *


*
Amazonbot
CCBot
Claude-User
Claude-SearchBot
DuckAssistBot
PerplexityBot
Perplexity-User
GrokBot
Google-InspectionTool
Mediapartners-Google
Slurp
Sogou web spider
Yahoo Ad Monitoring
Tag Inspector Spider
Adsbot-Google


*
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper


UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper
*


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Googlebot Twitterbot SemrushBot SiteAuditBot AhrefsBot BLEXBot AdsBot-Google AdsBot-Google-Mobile MJ12bot dotbot DataForSeoBot adsbot barkrowler SeekportBot AwarioRssBot AwarioSmartBot PetalBot ZoomBot Seobility TurnitinBot trendictionbot serpstatbot


AdsBot-Google-Mobile AdsBot-Google AdIdxBot *


*
Applebot-Extended
AliyunSecBot
Amazonbot
Arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
anthropic-ai
AudigentAdBot
BLEXBot
Bytespider
Buck
CCBot
coccocbot-web
cohere-ai
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
DataForSeoBot
DuckAssistBot
FacebookBot
ImageSift
JenkersBot
Jetslide
MoodleBot
meta-externalagent
Meta-ExternalAgent
Meta-ExternalFetcher
meta-externalfetcher
MyCentralAIScraperBot
Mojeek
news-please
OAI-SearchBot
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PetalBot
PerplexityBot
Perplexity-User
Poseidon Research Crawler
quillbot.com
Quora-Bot
Scrapy
Seekr
SeekrBot
SeznamBot
SeznamHomepageCrawler
scalepostAI
SentiOne
TaraGroup Intelligent Bot
Timpibot
TurnitinBot
ViennaTinyBot
yacy
YandexAdditional
YandexAdditionalBot
YouBot


* Mediapartners-Google Googlebot-Image cohere-ai magpie-crawler omgili omgilibot Perplexity-User PerplexityBot Applebot-Extended Bytespider CCBot DuckAssistBot


Yandex Baiduspider facebookexternalhit Yeti *


Googlebot Applebot Bingbot AdIdxBot baiduspider msnbot facebookexternalhit Slurp Yandex rogerbot dotbot *


ClaudeBot ULSTPropertyBot AdsBot-Google *


* Applebot Applebot-Extended NewsWhipbot


* Googlebot AdsBot-Google Googlebot-Image Bingbot Yandex DuckDuckBot FacebookBot Twitterbot AhrefsBot SemrushBot OAI-SearchBot ChatGPT-User PerplexityBot Claude-SearchBot GPTBot ClaudeBot CCBot


CCBot ChatGPT-User GPTBot Google-Extended Google-CloudVertexBot Omgilibot Omgili FacebookBot ClaudeBot Claude-SearchBot Claude-User Diffbot DuckAssistBot PerplexityBot Perplexity-User cohere-ai cohere-training-data-crawler Meta-ExternalAgent Meta-ExternalFetcher Meta-WebIndexer Timpibot Webzio-Extended YouBot Amazonbot Bytespider anthropic-ai OAI-SearchBot VelenPublicWebCrawler Gemini-Deep-Research ICC-Crawler MistralAI-User *


Orthogaffe
UbiCrawler
DOC
Zao
gsa-crawler
sitecheck.internetseer.com
Zealbot
AmazonAdBot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper
CNCDialer
Maxthon
MJ12bot
Slurp
*
AI2Bot
ChatGLM
ChatGLM-Spider
cohere-training-data-crawler
Diffbot
OmigiliBot
PetalBot
YouBot


*
AdsBot-Google
AdsBot-Google-Mobile
facebookexternalhit
AhrefsSiteAudit
AI2Bot
archive.org_bot
Botify
BotifyCrawler
BotifyTester
BrightEdge Crawler
claritybot
ContentKing
DeepCrawl
DotBot
ia_archiver
JetOctopus
Lumar
MJ12bot
NovaAct
OnCrawl
Operator
Optimizer
Rogerbot
RyteBot
SearchmetricsBot
SemrushBot
SemrushBot-BA
SemrushBot-CT
SemrushBot-FT
SemrushBot-OCOB
SemrushBot-SI
SemrushBot-SWA
SeobilityBot
SERankingBot
Serpstatbot
SISTRIX Crawler
SiteAuditBot
Sitebulb
SplitSignalBot
SurferBot
woorank


Crescent Internet ToolPak HTTP OLE Control v.1.0


* AmazonAdBot anthropic-ai AwarioSmartBot Bytespider CCBot ClaudeBot DataForSeoBot Diffbot newspaper peer39_crawler Scrapy Timpibot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* GPTBot ClaudeBot Google-Extended PerplexityBot


2checkout
AASA-Bot
AccessibleWebBot
Adagiobot
adbeat_bot
ADmantX
AdometryBot
Adsense-Snapshot-Google
AdsTxtCrawler
Adyen
AhrefsBot
AhrefsSiteAudit
Akismet
Algolia Crawler
AmazonProductDiscovery
Aolbot-News
APIs-Google
AppInsights
Applebot
AppleNewsBot
Archive-It
archive.org_bot
ArchiveBot
ArchiveTeam-ArchiveBot
AspiegelBot
Audisto Crawler
Awario
Baidu-ADS
Baiduspider
Barkrowler
bingbot
BingPreview
bitlybot
bl.uk_ldfc_bot
BLP_bbot
bordermanager
botify
BrandVerity
Bravebot
BrightEdge Crawler
BrightEdge-Crawler
BUbiNG
CaliperBot
CapitalOneBot
Catchpoint
cf-qualys-scanner
Chrome-Lighthouse
CJBot
ClarityBot
Clickagy Intelligence Bot v2
Cloudflare-AI-Search
CloudflareBrowserRenderingCrawler
coccocbot
ContextualBot
CookieReports
Coveobot
CriteoBot
curly-
cXensebot
Datadog
DataForSeoBot
Daum
demandbase
Discordbot
DotBot
DoubleVerifyBot
Doximity-Pipeline
Dragonbot
Dratabot
DuckAssistBot
DuckDuckBot
DuckDuckGo-Favicons-Bot
DVBot
eContext
ecurly Webhook
EngageBDR
europarchive.org
evidon
ExaleadCloudview
FacebookBot
facebookcatalog
facebookexternalhit
Facebot
FastDAST
FeedBurner
Feedfetcher-Google
FlipboardProxy
FlippBot
Freshping
GeedoProductSearch
Google Favicon
Google News
Google Page Speed Insights
Google Wireless Transcoder
Google-Apps-Script
Google-Certificates-Bridge
Google-CWS
Google-Display-Ads-Bot
Google-Firebase
Google-HotelAdsVerifier


* Googlebot-Image Googlebot-News Flipboard Amazonbot Applebot Applebot-Extended Bytespider CCBot ChatGPT-User Claude-SearchBot Claude-User Claude-Web ClaudeBot FacebookBot GPTBot Google-Extended Meta-ExternalAgent MistralAI-User OAI-SearchBot Perplexity-User PerplexityBot anthropic-ai cohere-ai cohere-training-data-crawler AhrefsBot


FAST Enterprise Crawler
FAST-WebCrawler
barkrowler
rogerbot
dotbot
DataForSeoBot
BLEXBot
AwarioRssBot
AwarioSmartBot
MJ12bot
ZoomBot
CCBot
Bytespider
*


* GPTBot CCBot Google-Extended anthropic-ai


* AmazonAdBot bingbot Yandex ias_crawler weborama-fetcher


* Google-Extended GPTBot CCBot FacebookBot Omgilibot


* grapeshot GPTBot ClaudeBot Google-Extended Amazonbot CCBot


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


AhrefsBot AhrefsBot/5.0 GPTBot * Teoma Mediapartners-Google Skynet The-Matrix Matrix HAL9000 GroundControl Ground-Control Carmine Caesar TCBear


AI2Bot Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai DeepSeek Diffbot Google-Extended GPTBot magpie-crawler omgili omgilibot PerplexityBot Google-CloudVertexBot meta-externalagent meta-externalfetcher *


* Bytespider CCBot claude-searchbot claude-user Claude-Web ClaudeBot FacebookBot Google-Extended MJ12bot NewsNow Perplexity-ai PerplexityBot PiplBot anthropic-ai cohere-ai ia_archiver magpie-crawler news-please omgili omgilibot meta-externalagent meta-externalfetcher A6-Indexer AhrefsBot Amazon-kendra Amazon-qbusiness amazonbot amzn-searchbot amzn-user Applebot-Extended archive.org_bot AwarioRssBot AwarioSmartBot BLP_bbot claritybot DataForSeoBot Diffbot EasouSpider ecoResearch FriendlyCrawler Genieo ImagesiftBot Meltwater Meta-ExternalAgent peer39_crawler peer39_crawler/1.0 PetalBot psbot R6_CommentReader Scrapy Seekr Spinn3r TurnitinBot WebVac WebZip FontReport


*
AI2Bot
anthropic-ai
Applebot-Extended
Arquivo-web-crawler
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
coccocbot-web
cohere-ai
cohere-training-data-crawler
Crawlspace
Diffbot
FacebookBot
FriendlyCrawler
Gensparkbot
Google-CloudVertexBot
Google-Extended
GPTBot
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
magpie-crawler
Meta-ExternalAgent
Meta-ExternalFetcher
Netvibes
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
Perplexity-ai
Perplexity-User
PetalBot
Scrapy
SemrushBot
Timpibot
Turnitin
TurnitinBot
VelenPublicWebCrawler
Webzio-Extended
yacy
yacybot
Yandex
YandexAdditional
YandexAdditionalBot
YouBot


Mediapartners-Google* AmazonAdBot * trendkite-akashic-crawler dotbot AhrefsBot DataForSeoBot SemrushBot


Twitterbot OAI-SearchBot ChatGPT-User PerplexityBot FirecrawlAgent AndiBot ExaBot PhindBot YouBot facebookexternalhit GPTBot CCBot Google-Extended *


* Googlebot Googlebot-News GoogleOther-Image GoogleOther-Video Bingbot DuckDuckBot Slurp Applebot facebookexternalhit Meta-ExternalAgent Meta-ExternalFetcher SmartNews-AdsBot SmartNewsBot Twitterbot Applebot-Extended Bytespider CCBot Claude-Web ClaudeBot Diffbot FriendlyCrawler GPTBot Google-Extended GoogleOther ICC-Crawler ImagesiftBot PerplexityBot PetalBot Scrapy Timpibot VelenPublicWebCrawler Webzio-Extended YouBot anthropic-ai cohere-ai img2dataset omgili omgilibot


Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/w4mwnpbXf3MFAbxOkJRw; +http://www.echobox.com)

Mozilla/5.0 (compatible; NewsRoom.BI/0.1; +http://www.newsroom.bi/bot.html)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36(KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; mrfCompass-Marshall/1.0)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Jukebox/1.0)


* AdsBot-Google AdsBot-Mobile-Google Bingbot AdIdxBot


Pinterestbot FacebookBot facebookexternalhit/1.1 facebookexternalhit/1.0 bingbot Googlebot Googlebot-Image AdsBot-Google-Mobile AdsBot-Google AdsBot-Google-Mobile-Apps *


* CazoodleBot MJ12bot dotbot/1.0 Gigabot Mediapartners-Google AdsBot-Google Googlebot-Image Googlebot Bingbot Baiduspider GPTBot PerplexityBot ClaudeBot Google-Extended YandexBot Qwantify SeznamBot Yeti Slurp


* adsbot-google Nutch AhrefsBot AhrefsSiteAudit MJ12bot Pinterest


Amazonbot
Applebot
GPTBot
A6-Indexer
AhrefsBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot-Extended
Baiduspider
betaBot
BLEXBot
botify
brands-bot-logo
Brightbot
Bytespider
calculon
CCBot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
collection@infegy.com
Cotoyogi
Crawl4AI
CrunchBot
deepcrawl
DeepSeekBot
Diffbot
DirBuster
DTS
DuckAssistBot
EasouSpider
FacebookBot
FirecrawlAgent
FriendlyCrawler
GlutenCrawler
Google-CloudVertexBot
Google-Extended
GrapeshotCrawler
iaskspider
ICC-Crawler
ImagesiftBot
img2dataset
JackCrawl
Jetslide
linkapedia_bot
Mediatoolkitbot
MegaIndex
meta-externalagent
meta-externalfetcher
Microsoft BITS
Microsoft URL Control
Microsoft-WebDAV-MiniRedir
MistralAI-User
news-please
Novellum
NTENTbot
Nutch
omgili
Omgilibot
OpenLinkProfiler
Perplexity-User
PerplexityBot
PetalBot
Primalbot
python-requests
Re-Animator
Riddler
SafeDNSBot
SBIntuitionsBot
Scrapy
SearchAtlas
SemanticScholarBot
SemrushBot
SentiBot
Sogou
Spider-Pig
sqlmap
StorygizeBot
Test Certificate Info
TikTokSpider
Timpibot
Turnitin
uipbot
vBSEO
VegeBot
VelenPublicWebCrawler
Web Link Validator
WebCruiser


* SemrushBot DotBot BLP_bbot YandexBot archive.org_bot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Yandex FacebookBot GrapeshotCrawler BLEXBot AdsrvrBot SemrushBot PingdomBot MJ12bot Twiceler dotbot Sogou Baidu Baiduspider YandexMarket yaanibot Adsbot-Google


* ia_archiver archive.org_bot heritrix My-heritrix-crawler Timpibot omgili Bytespider DuckAssistBot Applebot-Extended Google-Extended Google-CloudVertexBot GoogleOther GPTBot CCBot Amazonbot FacebookBot cohere-ai cohere-training-data-crawler Omgilibot ClaudeBot Diffbot PanguBot MistralAI-User webz.io DeepSeekBot AmazonAdBot Amzn-SearchBot Amzn-User Claude-SearchBot Claude-User PetalBot Meta-ExternalFetcher Meta-WebIndexer OAI-SearchBot ChatGPT-User PerplexityBot Perplexity-User


* AdsBot-Google AdsBot-Google-Mobile AdIdxBot Google-Extended Gemini-Deep-Research GPTBot ChatGPT-User OAI-SearchBot PerplexityBot Perplexity-User ClaudeBot Claude-User Claude-SearchBot AhrefsBot dotbot MJ12bot PetalBot Yandex


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
ChatGPT-user
CCBot
GPTBot
Google-Extended
ImagesiftBot
FacebookBot
Omgilibot
Amazonbot
anthropic-ai
Claude-Web
cohere-ai
omgili
PerplexityBot
YouBot
Bytespider
Diffbot
Applebot-Extended
OAI-SearchBot
ClaudeBot
Google-CloudVertexBot
Meta-ExternalAgent
Meta-ExternalFetcher
Quora-Bot
DataForSeoBot
Webzio-Extended
FriendlyCrawler
ChatGPT-User
meta-externalagent
meta-externalfetcher
TeleportPro
Qwam content intelligence
Timpibot
Kangaroo Bot
AI2Bot
VelenPublicWebCrawler
img2dataset
Sidetrade indexer bot
PanguBot
ISSCyberRiskCrawler
Ai2Bot-Dolma
Claude-SearchBot
Claude-User
MistralAI-User
DeepSeekBot
Perplexity-User


facebookexternalhit AmazonAdBot * Googlebot-News Google-Extended


* Googlebot Bingbot Slurp DuckDuckBot OAI-SearchBot PerplexityBot ChatGPT-User Claude-Web AmazonAdBot Amazonbot anthropic-ai Applebot-Extended archive.org_bot Arquivo-web-crawler AwarioRssBot AwarioSmartBot Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler DataForSeoBot Diffbot DuckAssistBot europarchive.org FacebookBot FriendlyCrawler Google-CloudVertexBot heritrix ia_archiver ia_archiver-web.archive.org ImagesiftBot LinkArchiver magpie-crawler Meta-ExternalAgent meta-externalagent Meta-ExternalFetcher meta-externalfetcher news-please Nicecrawler omgili omgilibot peer39_crawler peer39_crawler/1.0 Primalbot Quora-Bot Scrapy Timpibot TurnitinBot web-archive-net.com.bot YouBot GPTBot Google-Extended


anthropic-ai
AmazonBot
Claude
Claude-Web
claudeweb
Amzn-SearchBot
Amzn-User
AndiBot
cohere-ai
ClaudeBot
ChatGPT-User
Googlebot-News
GPTBot
CCBot
Google-Extended
DataForSeoBot
Diffbot
Diffbot-user
Kagibot
FacebookBot
magpie-crawler
NewsNow
news-please
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Scrapy
TurnitinBot
Applebot-Extended
Meta-ExternalAgent
meta-externalagent
OAI-SearchBot
Bytespider
Claude-User
Claude-SearchBot
AI2Bot
AhrefsBot
Ai2Bot-Dolma
Applebot
Arquivo-web-crawler
AspiegelBot
AwarioRssBot
AwarioSmartBot
Barkrowler
coccocbot-web
cohere-training-data-crawler
crawler4j
Crawlspace
dotbot
DuckAssistBot
FriendlyCrawler
Google-CloudVertexBot
https://hada.news
https://www.imediaethics.org
iaskspider/2.0
ICC-Crawler
icc-crawler
ICC Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
LivelapBot
MauiBot
Meltwater
Meta-ExternalFetcher
MJ12bot
Mojeek
MojeekBot
MoodleBot
NeticleBot
Netvibes
PanguBot
PetalBot
SeekportBot
Seekr
SemrushBot
SeznamBot
Sidetrade Indexer Bot
Timpibot
Turnitin
VelenPublicWebCrawler
Webzio-Extended
yacy
yacybot
Yandex
YandexAdditional
YandexAdditionalBot
YandexBot
YandexImages
YandexNews
Baiduspider
Baiduspider-image
BaiduSpider-render
Baiduspider-video
Sogou Web Spider
yisouspider


GPTBot PerplexityBot Google-Extended anthropic-ai Claude-Web *


bingbot
*
FacebookExternalHit
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
Brightbot 1.0
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Diffbot
DuckAssistBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
imgproxy
ISSCyberRiskCrawler
Kangaroo Bot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
NovaAct
OAI-SearchBot
omgili
omgilibot
Operator
PanguBot
Perplexity-User
PerplexityBot
PetalBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
TikTokSpider
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot


*
Google-CloudVertexBot
Google-Extended
ChatGPT-User
GPTBot
OAI-SearchBot
OAISearch
anthropic-ai
claude-web
claudebot
facebookbot
meta-externalagent
meta-externalfetcher
applebot-extended
perplexitybot
cohere-ai
youbot
amazonbot
aliyunsecbot
audigentadbot
awariorssbot
awariosmartbot
blexbot
bytespider
ccbot
dataforseobot
diffbot
duckassistbot
echoboxbot
friendlycrawler
ia_archiver
imagesiftbot
mycentralaiscraperbot
newsnow
news-please
omgili
omgilibot
webzio-extended
peer39_crawler
peer39_crawler/1.0
quillbot.com
quora-bot
scrapy
seekrbot
seznamhomepagecrawler
taragroup intelligent bot
timpibot
turnitinbot
viennatinybot
jetslide
magpie-crawler
poseidon research crawler


Mediapartners-Google
Google-Display-Ads-Bot
AdsBot-Google
AdsBot-Google-Mobile
TwitterBot
Adidum
Bloglines/3.1
DOC
Download Ninja
Exabot
Fetch
HTTrack
Jyxobot/1
Mail.Ru
Microsoft.URL.Control
NPBot
Offline Explorer
SiteSnagger
Speedy
Teleport
TeleportPro
UbiCrawler
WebCopier
WebReaper
WebStripper
WebZIP
Xenu
Zao
Zealbot
ZyBORG
cityreview
dotbot
e-SocietyRobot
grub-client
larbin
libwww
linko
psbot
sitecheck.internetseer.com
wget
*


* grapeshot GPTBot ClaudeBot Google-Extended Amazonbot CCBot


waybackurls AdsBot-Google Googlebot AdsBot-Google-Mobile *


* OpenAI ChatGPT-User Google-Extended OpenAI-Crawler GPTBot OAI-SearchBot


* GPTBot ChatGPT-User OAI-SearchBot Google-Extended PerplexityBot ClaudeBot Applebot-Extended Meta-ExternalAgent Meta-ExternalFetcher CCBot Amazonbot cohere-ai Omgilibot Omgili


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* AdsBot-Google Googlebot Googlebot-Image


* GPTBot Bytespider Amazonbot ClaudeBot BLP_bbot Slackbot


Baiduspider BaiduImagespider BaiduMobaider Yeti Googlebot *


* GPTBot NewsNow CCBot TurnitinBot PetalBot MoodleBot BLEXBot R6_CommentReader ICC-Crawler TinEye-bot Nutch trendkite-akashic-crawler


*
OAI-SearchBot
GPTBot
ChatGPT Agent
ChatGPT-User
ClaudeBot
Claude-Web
Claude-User
Claude-SearchBot
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GoogleAgent-Mariner
Google-CloudVertexBot
CloudVertexBot
Meta-ExternalAgent
Meta-ExternalFetcher
meta-externalagent
meta-externalfetcher
PerplexityBot
Perplexity-User
Applebot
Applebot-Extended
MistralAI-User
MistralAI-User/1.0
cohere-ai
cohere-training-data-crawler
YouBot
PhindBot
Amazonbot
Diffbot
Bytespider
CCBot


* GPTBot ChatGPT-User CCBot anthropic-ai TextBulkerBot


* AmazonAdBot GPTBot PerplexityBot ClaudeBot peer39_crawler Applebot-Extended Bytespider CCBot Google-Extended Meta-ExternalAgent meta-externalagent


Amazonbot
Cohere-AI
Bytespider
cohere-ai
Ai2Bot-Dolma
Timpibot
Omgilibot
Mistral AI-User
Amzn-User
diffbot-user
Amzn-SearchBot
DuckAssistBot
YouBot
seodiffbot
PetalBot
img2dataset
omgili
Scrapy
DeepSeekBot
Sogou web spider
Baiduspider


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* OAI-SearchBot GPTBot ChatGPT-User ClaudeBot PerplexityBot FirecrawlAgent AndiBot ExaBot PhindBot YouBot


* RavenCrawler rogerbot dotbot SemrushBot SiteAuditBot SplitSignalBot PowerMapper Swiftbot lyticsbot Dubbotbot PopeTech-CrawlBot PopeTech-ScanBot


* AmazonAdBot anthropic-ai ChatGPT-User ClaudeBot Claude-Web cohere-ai GPTBot OAI-SearchBot quillbot.com PerplexityBot TurnitinBot


* AhrefsBot Semrushbot Yeti GPTBot OAI-SearchBot PerplexityBot Google-Extended ClaudeBot Claude-SearchBot meta-externalagent Applebot-Extended CCBot


HubSpot
Bytespider
GPTBot
claudebot
panscient
PerplexityBot
ImagesiftBot
Amazonbot
Applebot-Extended
CCBot
Diffbot
FacebookBot
omgili
carbon-umbrella-bot
meta-externalagent
Opebot-v
OAI-SearchBot
ChatGPT-User
TaraGroup Intelligent Bot
SynthesiBot
AliyunSecBot
Baiduspider
Brightbot
Thinkbot
Sogou web spider
*


*
GPTBot
ChatGPT-User
anthropic-ai
ClaudeBot
CCBot
Amazonbot
Bytespider
Applebot
Sogou web spider
Google-Extended


*
Ai2Bot-Dolma
AI2Bot
anthropic-ai
Applebot-Extended
Arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
Barkrowler
Bytespider
CCBot
ClaudeBot
cohere-ai
cohere-training-data-crawler
crawler4j
dotbot
Diffbot
FacebookBot
FriendlyCrawler
GPTBot
https://hada.news
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MJ12bot
PanguBot
Scrapy
Seekr
Timpibot
VelenPublicWebCrawler
Webzio-Extended


*
Googlebot
AliyunSecBot
AmazonBot
anthropic-ai
Applebot-Extended
archive.org_bot
AudigentAdBot
AwarioRssBot
AwarioSmartBot
BLEXBot
Bytespider
CCBot
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
cohere-ai
DataForSeoBot
Diffbot
DuckAssistBot
EchoboxBot
FacebookBot
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
GPTBot
ImagesiftBot
Jetslide
magpie-crawler
Meta-ExternalAgent
meta-externalagent
Meta-ExternalFetcher
meta-externalfetcher
Meta-WebIndexer
meta-webindexer/1.1
MyCentralAIScraperBot
NewsNow
news-please
OAI-SearchBot
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Perplexity-User
Poseidon Research Crawler
quillbot.com
Quora-Bot
Scrapy
SeekrBot
SeznamHomepageCrawler
TaraGroup Intelligent Bot
Timpibot
TurnitinBot
ViennaTinyBot
YouBot
AmazonAdBot
facebookexternalhit
Twitterbot
RedditBot