*
ImagesiftBot
AdsBot-Google
Mediapartners-Google
Google-Display-Ads-Bot
AmazonAdBot
ias_crawler
ias_wombles
proximic
TTD-Content
Quantcastbot
CriteoBot/0.1
InteractiveAdvertisingBureau
Leikibot
GrapeshotCrawler
adidxbot
Yahoo Ad Monitoring
peer39_crawler
Connatix/FeedCrawler
Connatix/WebCrawler
bidswitchbot
Verity
GumGumAdsTxtCrawler/1.0
IAB-Tech-Lab
TheCoolCompany-Crawler
SPM-AdsTxt-Verifier
OpenXAdsTxtCrawler
PubMatic Crawler Bot
Seedtag-Node-Request
adstxt/1.0
Jounce
Kargo Ads.txt Crawler


* GPTBot ChatGPT-User Google-Extended CCBot


AI2Bot Ai2Bot-Dolma aiHitBot Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler DuckAssistBot FacebookBot Google-Extended GoogleOther GoogleOther-Image GoogleOther-Video GPTBot img2dataset Meta-ExternalAgent MyCentralAIScraperBot omgili omgilibot Quora-Bot TikTokSpider YouBot AdsBot-Google AdsBot-Google-Mobile AdsBot-Google-Mobile-Apps *


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* GPTBot ChatGPT-User Google-Extended ClaudeBot Claude-Web FacebookBot CCBot SemrushBot AhrefsBot MJ12bot DotBot PetalBot BLEXBot


Yandex YandexTurbo YandexBot * 008 msnbot bingbot Pinterest


Yandex * BLEXBot MJ12bot SemrushBot PetalBot CriteoBot proximic grapeshot ia_archiver


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


*
Baiduspider
AddSearchBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
amazon-kendra
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Amzn-User
Andibot
Anomura
anthropic-ai
Applebot
Applebot-Extended
atlassian-bot
Awario
AzureAI-SearchBot
bedrockbot
bigsur.ai
Bravebot
Brightbot 1.0
BuddyBot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Crawlspace
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
Google-Firebase
Google-NotebookLM
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iAskBot
iaskspider
iaskspider/2.0
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
img2dataset
ISSCyberRiskCrawler
kagi-fetcher
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
LAIONDownloader
LCC
LinerBot
Linguee Bot
LinkupBot
Manus-User
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
meta-webindexer
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NotebookLM
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
Operator


* Googlebot Googlebot-Image MediaPartners-Google AdsBot-Google-Mobile AdsBot-Google Facebot facebookexternalhit OAI-SearchBot ChatGPT-User GPTBot


* GPTBot Google-Extended anthropic-ai CCBot Omgilibot cohere-ai ChatGPT-User OAI-SearchBot PerplexityBot Amazonbot YouBot Claude-Web Claudebot Claude-User Claude-SearchBot Diffbot Bytespider ImagesiftBot Meltwater Seekr


bingbot coccocbot-web AhrefsBot YandexBot * Googlebot Twitterbot Amazonbot Anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider Barkrowler BLEXBot CCBot ChatGPT-User ClaudeBot Claude-Web Cohere-ai DataForSeoBot DotBot FacebookBot GPTBot ImagesiftBot Magpie-crawler MJ12bot Omgili Omgilibot Peer39_crawler Peer39_crawler/1.0 PerplexityBot YouBot YaK Y!J-ASR SeznamBot PetalBot SemrushBot QuantBot Qwantbot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent Yandex


archive.org_bot bingbot DuckDuckBot Googlebot Googlebot-Mobile ia_archiver Yandex Adsbot-Google AhrefsBot Googlebot-Image Mail.Ru Mediapartners-Google msnbot msnbot-media Slurp StackRambler Yahoo-MMCrawler *


*
Twitterbot
facebookexternalhit
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Andibot
anthropic-ai
Applebot
Applebot-Extended
bedrockbot
Brightbot 1.0
Bytespider
CCBot
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Diffbot
DuckAssistBot
EchoboxBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
MistralAI-User/1.0
MyCentralAIScraperBot
NovaAct
OAI-SearchBot
omgili
omgilibot
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot
SemrushBot-BA
SemrushBot-CT
SemrushBot-OCOB
SemrushBot-SI
SemrushBot-SWA
Sidetrade indexer bot
TikTokSpider
Timpibot
VelenPublicWebCrawler
Webzio-Extended
wpbot
YandexAdditional
YandexAdditionalBot
YouBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


"008"
AddSearchBot
AI2Bot
Ai2Bot-DeepResearchEval
Ai2Bot-Dolma
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-extended
atlassian-bot
AutoRAG
AwarioSmartBot
AzureAI-SearchBot
Big Sur AI
bigsur.ai
Brandwatch
Bravebot
Brightbot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DDM-DCipher*
DDM-DCipher/1.0.7
DDM*
DeepSeekBot
Devin
Diffbot
Direqt Anomura
DuckAssistBot
Echobot Bot
ExaBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
GPTBot
HTTrack
iAsk
iAskBot
iaskspider
ICC Crawler
ICC-Crawler
ImageSiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
Kunato
laion-huggingface-processor
LCC
LINER Bot
LinerBot
LinkupBot
Manus-User
meta-externalagent
meta-externalfetcher
meta-webindexer
mistral.ai
MistralAI-User
netEstate Imprint Crawler
NovaAct
Novellum AI Crawl
Nutch
OAI-SearchBot
Offline Explorer
omgili
opencode
PanguBot
PeopleInc-DCipher-Scraper/1.1.0
Perplexity-User
PerplexityBot
PetalBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
Scrapy
SeekrBot
SemrushBot-OCOB
SemrushBotSwa
ShapBot
Spider
TavilyBot
TerraCotta
Timpibot
Trae
TwinAgent


*
moget
ichiro
NaverBot
Yeti
Baiduspider
Baiduspider-video
Baiduspider-image
sogou spider
YoudaoBot
YandexBot/3.0
YandexBot
http://yandex.com/bots
Yandex
CrystalSemanticsBot
PetalBot


rogerbot dotbot GPTBot OpenAI CCBot ClaudeBot PerplexityBot *


Crescent Internet ToolPak HTTP OLE Control v.1.0


* AdsBot-Google AdsIdxBot Google-InspectionTool Twitterbot facebookexternalhit LinkedInBot


* AhrefsBot MJ12bot SemrushBot-SA SemrushBot-BA SemrushBot-SI SemrushBot-SWA SemrushBot-CT SemrushBot-BM


claritybot GPTBot Google-Extended CCBot ChatGPT-User anthropic-ai Omgilibot Omgili FacebookBot Bytespider *


proximic dotbot Barkrowler trendictionbot SirdataBot ICC-Crawler meta-externalagent bingbot ClaudeBot peer39_crawler peer39_crawler/1.0 *


*
bingbot
GoogleOther
GoogleOther-Image
GoogleOther-Video
APIs-Google
The Googlebot-News
Googlebot smartphone
AdsBot-Google
Googlebot
Googlebot-Image
Google-InspectionTool
Google Page Speed
Chrome-Lighthouse
Googlebot-Video
Storebot-Google
AdsBot-Google-Mobile
Mediapartners-Google
developers.google.com/*
W3C_Validator
SEMrush Crawler
Screaming Frog SEO Spider


* Googlebot Yandex GPTBot PerplexityBot Google-Extended ClaudeBot


*
Scrappy
Scrappy Bot
discobot
dotbot
yacybot
Adsbot-Google
AdsBot-Google-Mobile


* Googlebot-News Googlebot Bingbot Applebot PetalBot Bytespider Amazonbot


MJ12bot GPTBot/1.0 GPTBot* SemrushBot* DataForSeoBot* SeznamBot* ImagesiftBot AhrefsBot* ClaudeBot* Amazonbot*


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* Googlebot Yeti Daum Daumoa Ads-Naver facebookexternalhit


* psbot magpie-crawler turnitinbot Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-Web cohere-ai Diffbot DuckAssistBot FriendlyCrawler Gemini-Deep-Research Google-CloudVertexBot Google-Extended GoogleOther GPTBot ICC-Crawler img2dataset MistralAI-User OAI-SearchBot omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot Perplexity-User Webz.io YouBot facebookexternalhit Applebot bingbot msnbot Yandex Wget


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent AdsBot-Google-Mobile AdsBot-Google PetalBot dotbot AhrefsBot


*
Googlebot
Googlebot-Image
Googlebot-News
Googlebot-Video
Mediapartners-Google
AdsBot-Google
Bingbot
MSNBot
BingPreview
AdIdxBot
Slurp
Baiduspider
Baiduspider-image
Baiduspider-video
Baiduspider-news
Yandex
YandexBot
YandexImages
YandexVideo
YandexNews
YandexBlogs
YandexMedia
DuckDuckBot
Sogou
Sogou web spider
Sogou Pic Spider
Sogou head spider
Sogou Orion spider
Sogou-Test-Spider
SeznamBot
Yeti
Exabot
Twiceler
ia_archiver
Twitterbot
facebot
facebookexternalhit
facebookcatalog
meta-externalagent
meta-externalfetcher
GPTBot
ChatGPT-User
CCBot
ClaudeBot
anthropic-ai
cohere-ai
Writer
AhrefsBot
AhrefsSiteAudit
SemrushBot
DotBot
BLEXBot
Barkrowler
MJ12bot
SerpstatBot
MozDotBot
Screaming Frog SEO Spider
DomainStatsBot
ZoomBot
SiteCheckerBot
Cliqzbot
SeobilityBot
UptimeRobot
Google-Structured-Data-Testing-Tool
Google-Read-Aloud
Google-InspectionTool
sqlmap
sqlmap/1.0-dev
Acunetix
masscan
Nessus
CensysInspect
AlphaBot
Go-http-client
Offline Explorer
WebCopier
Scrapy
Nutch
Heritrix
BacklinkCrawler
008
DataForSeoBot
PetalBot
Seekport Crawler
Zoominfobot
ZoominfoBot
CrunchBot
omgili
Python-urllib
python-requests
libwww-perl
httpx
curl
Wget
OkHttp
Java
TurnitinBot
crawler4j
OpenLinkProfiler


*
AwarioRssBot
AwarioSmartBot
Baiduspider
Barkrowler
BLEXBot
CazoodleBot
DataForSeoBot
DotBot
dotbot/1.0
Gigabot
Go-http-client
Linespider
magpie-crawler
MojeekBot
Node/simplecrawler
Nuclei
Perplexity-ai
Perplexity-User
PerplexityBot
PetalBot
PiplBot
ProRataInc
Riddler
Scrapy
SeznamBot
Sidetrade indexer bot
Sogou Spider
Timpibot
TurnitinBot
VelenPublicWebCrawler
Webzio-Extended
WikiDo
Yandexbot
YouBot
Zoominfobot
archive.is
archive.today
archive.ph


ChatGPT-User Perplexity-User Claude-User GPTBot PerplixityBot ClaudeBot anthropic-ai Google-Extended Applebot-Extended CCBot Meta-ExternalAgent FacebookBot Bytespider DeepSeek Diffbot ImagesiftBot cohere-ai Yandex YandexBot Baiduspider Baiduspider-video Baiduspider-image *


* Googlebot Bingbot YandexBot DuckDuckBot SemrushBot AhrefsBot MJ12bot DotBot BLEXBot *bot*


* GPTBot ClaudeBot Claude-User Google-Extended CCBot Bytespider Amazonbot anthropic-ai PerplexityBot YouBot


* grapeshot YandexBot SentiBot AhrefsBot MJ12bot Exabot BLEXBot DotBot SemrushBot bingbot


*
AhrefsBot
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot-Extended
AwarioBot
AwarioRssBot
AwarioSmartBot
Bravebot
Bytespider
CCBot
ClaudeBot
Crawl4AI
cohere-ai
cohere-training-data-crawler
Cotoyogi
DataForSeoBot
Dataprovider Spider
DeepSeekBot
diffbot
DotBot
Factset_spyderbot
Facebookbot
FirecrawlAgent
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
Google-NotebookLM
GoogleOther
GoogleOther-Extended
GPTBot
iaskspider
ICC-Crawler
aiHitBot
img2dataset
ImagesiftBot
kagi-fetcher
Kangaroo Bot
magpie-crawler
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
meta-webindexer
MistralAI-User
MJ12bot
NovaAct
Omgili
Omgilibot
Operator
PanguBot
PhindBot
peer39_crawler
PiplBot
Qwen-LLMAgent
scoop.it
Scrapy
Screaming Frog SEO Spider
Seekr
Sentibot
SemrushBot
SemrushBot-OCOB
Sidetrade indexer bot
Timpibot
TikTokSpider
TurnitinBot
VelenPublicWebCrawler
webzio-extended
WRTNBot
ZoominfoBot
Baiduspider
YandexBot
YandexImages
PetalBot
360Spider
Mail.RU_Bot
backdorbot
BlackWidow
FlashGet
GetRight
python
urllib
Go-http-client
ChatGPT-User
PerplexityBot
DuckAssistBot
YouBot
OAI-SearchBot
Claude-User
Claude-SearchBot


Mediapartners-Google
AmazonBuyForMe
Amzn-User
ChatGPT Agent
ChatGPT-User
Claude-User
Claude-Web
Devin
Gemini-Deep-Research
Google-NotebookLM
GoogleAgent-Mariner
Manus-User
MistralAI-User
MistralAI-User/1.0
NotebookLM
NovaAct
Operator
Perplexity-User
TwinAgent
AddSearchBot
atlassian-bot
Echobox Bot
EchoboxBot
facebookexternalhit
meta-externalfetcher
Meta-ExternalFetcher
WARDBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
anthropic-ai
Applebot-Extended
bedrockbot
Bytespider
CCBot
ChatGLM-Spider
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
DeepSeekBot
FacebookBot
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
GPTBot
img2dataset
laion-huggingface-processor
LAIONDownloader
meta-externalagent
Meta-ExternalAgent
meta-webindexer
PanguBot
Timpibot
VelenPublicWebCrawler
Andibot
AzureAI-SearchBot
Bravebot
Claude-SearchBot
DuckAssistBot
iAskBot
iaskspider
iaskspider/2.0
IbouBot
kagi-fetcher
LinkupBot
LinerBot
OAI-SearchBot
PerplexityBot
PetalBot
PhindBot
TavilyBot
WRTNBot
YouBot
ZanistaBot
aiHitBot
amazon-kendra
Amazonbot
Amzn-SearchBot
Anomura
Applebot
Awario
bigsur.ai
Brightbot 1.0
BuddyBot
Channel3Bot
Cloudflare-AutoRAG
Crawl4AI
Crawlspace
Datenbank Crawler
Diffbot
Factset_spyderbot
FirecrawlAgent
ICC-Crawler
ImagesiftBot
imageSpider
ISSCyberRiskCrawler
Kangaroo Bot
KlaviyoAIBot


Googlebot
googlebot-image
googlebot-mobile
MSNBot
Slurp
yahoo-mmcrawler
Twitterbot/1.0
TwitterBot 1.0
AhrefsBot
AhrefsSiteAudit
Screaming Frog
Screaming Frog SEO Spider
yahoo-blogs/v3.9
Bingbot
*
"008"
Amazonbot
anthropic-ai
Applebot
Applebot-extended
Bytespider
CCBot
ClaudeBot
Claude-Web
cohere-ai
Diffbot
FacebookBot
GPTBot
HTTrack
Nutch
Offline Explorer
omgili
Scrapy
YouBot
meta-externalagent
Timpibot
DuckAssistBot
Perplexity-User
Ai2Bot-Dolma
meta-externalfetcher
ChatGPT-User
PerplexityBot
OAI-SearchBot


* ADmantX AlphaBot anthropic-ai AwarioRssBot AwarioSmartBot BLEXBot Buzzbot Bytespider CCBot ChatGPT-User claritybot Claude-Web ClaudeBot cohere-ai Diffbot FacebookBot FriendlyCrawler Google-Extended GPTBot huggingface ImagesiftBot img2dataset magpie-crawler Meltwater Neevabot news-please NewsNow Nutch omgili omgilibot panscient.com Perplexity-ai PerplexityBot PetalBot PiplBot scoop.it Scrapy Seekr SentiBot SeznamBot TurnitinBot YouBot ZumBot


*
DMCAAgent
CopyrightCrawler
AntiPiracyBot
008
Slurp
goo-crawler
RakutenBot
Teoma
Baiduspider
Sogou Spider
360Spider
YisouSpider
ia_archiver
MusoBot
Link-BusterBot


* Mediapartners-Google trovitBot pricebot magpie-crawler SemrushBot trendictionbot BLEXBot


claritybot GPTBot Google-Extended CCBot ChatGPT-User anthropic-ai Omgilibot Omgili FacebookBot Bytespider *


Mediapartners-Google YisouSpider Leikibot ShopWiki NPBot psbot HenryTheMiragoRobot gigabot appie Meta-ExternalAgent AhrefsBot *


*
Ai2Bot-Dolma
AI2Bot
anthropic-ai
Applebot-Extended
Arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
Barkrowler
Bytespider
CCBot
ClaudeBot
cohere-ai
cohere-training-data-crawler
crawler4j
dotbot
Diffbot
FacebookBot
FriendlyCrawler
GPTBot
https://hada.news
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MJ12bot
PanguBot
Scrapy
Seekr
Timpibot
VelenPublicWebCrawler
Webzio-Extended


* CCBot GPTBot ChatGPT-User Google-Extended anthropic-ai cohere-ai PerplexityBot Applebot-Extended Grok OAI-SearchBot Perplexity-ai Perplexity-User


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


*
Googlebot-Image
360Spider
404checker
404enemy
80legs
Abonti
Aboundex
Aboundexbot
Acunetix
ADmantX
adscanner
AfD-Verbotsverfahren
AhrefsBot
AIBOT
AiHitBot
Aipbot
Alexibot
Alligator
AllSubmitter
AlphaBot
Anarchie
Apexoo
archive.org_bot
ASPSeek
Asterias
Attach
autoemailspider
BackDoorBot
Backlink-Ceck
backlink-check
BacklinkCrawler
BackStreet
BackWeb
Badass
Bandit
Barkrowler
BatchFTP
Battleztar Bazinga
BBBike
BDCbot
BDFetch
BetaBot
Bigfoot
Bitacle
Blackboard
Black Hole
BlackWidow
BLEXBot
Blow
BlowFish
Boardreader
Bolt
BotALot
Brandprotect
Brandwatch
Buddy
BuiltBotTough
BuiltWith
Bullseye
BunnySlippers
BuzzSumo
Calculon
CATExplorador
CazoodleBot
CCBot
Cegbfeieh
CheeseBot
CherryPicker
CheTeam
ChinaClaw
Chlooe
Claritybot
Cliqzbot
Cloud mapping
coccocbot-web
Cogentbot
cognitiveseo
Collector
com.plumanalytics
Copier
CopyRightCheck
Copyscape
Cosmos
Craftbot
crawler4j
crawler.feedback
crawl.sogou.com
CrazyWebCrawler
Crescent
CrunchBot
CSHttp
Curious
Custo
DatabaseDriverMysqli
DataCha0s
DBLBot
demandbase-bot
Demon
Deusu


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


*
Amazonbot
AlibabaBot
Ai2Bot-Dolma
Applebot-Extended
Bytespider
CCBot
ChatGLM-Spider
ClaudeBot
Claude-Web
CloudVertexBot
cohere-training-data-crawler
Cotoyogi
Datenbank Crawler
Diffbot
FacebookBot
Google-Extended
GoogleOther
GPTBot
ICC-Crawler
imageSpider
Kangaroo Bot
laion-huggingface-processor
LCC
meta-externalagent
netEstate Imprint Crawler
omgili
PanguBot
PerplexityBot
SBIntuitionsBot
Spider
Timpibot
VelenPublicWebCrawler
webzio-extended
YouBot
008
Dataprovider.com
dcrawl
HelloworkJobPostingBot
HTTrack
HTTrack 3.0
KrawlerBot
MetaInspector
MetaJobBot
newspaper
Nutch
Offline Explorer
OpenindexSpider
Potions
Scrapy
ServerHunterSpider
StatsDroneBot
CloudflareBrowserRenderingCrawler
aa
AhrefsBot
Attracta
Barkrowler
BrightEdge Crawler
BLEXBot
CaliberBot
Caliperbot
ClarityBot
Clearscopebot
Cloudtrellis
cludo.com
Cocolyzebot
cognitiveSEO Crawler
contentking
Convermax
Cxense
DataForSeoBot
DataForSEO Bot
DomainStatsBot
DotBot
Dragonbot
Huckabot
Huckabuy Bot
hypestat
iaskspider
img2dataset
James BOT
LinkCheck by Siteimprove
LinkChecker Bot
linkchecker.pro
linkdexbot
LinksIndexerBot
MarketGoo
MBCrawler
MegaIndex.ru
MJ12bot
Moz dotbot
Moz rogerbot
Nitro-
NitroBot
online-webceo-bot
Prerender
Pro Sitemaps
Readable
RevvimGort
rogerbot-crawler


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* AhrefsBot SemrushBot MJ12bot DotBot BLEXBot PetalBot Bytespider CCBot


AhrefsBot YandexBot BLEXBot Qwantify ia_archiver *


Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot Google-Extended GPTBot meta-externalagent PetalBot TikTokSpider *


* GPTBot ChatGPT-User Google-Extended PerplexityBot Amazonbot ClaudeBot Omgilibot FacebookBot Applebot anthropic-ai Bytespider Claude-Web Diffbot ImagesiftBot Omgili YouBot


DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)


AhrefsBot
BUbiNG
dotbot
mj12bot
MJ12bot
deepcrawl
Scrapy
grub-client
k2spider
WebReaper
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
*


MistralAI-User Claude-User Applebot GPTBot ChatGPT-User CCBot Google-Extended anthropic-ai Claude-Web ClaudeBot Bytespider Amazonbot OAI-SearchBot PerplexityBot Applebot-Extended cohere-ai cohere-training-data-crawler FacebookBot Meta-ExternalAgent AhrefsBot *


Bytespider GPTBot claudebot panscient PerplexityBot ImagesiftBot Amazonbot Applebot-Extended CCBot Diffbot FacebookBot omgili


Baiduspider
Sosospider
sogou spider
YodaoBot
360spider
Bytespider
Bingbot
*


*
cohere-ai
YouBot
SeekportBot
ZoominfoBot
Barkrowler
MegaIndex
SeznamBot
Diffbot
CCBot
Linespider
dotbot
PetalBot
Bytespider
Sogou web spider
Sogou inst spider
Baiduspider
CriteoBot


Mediapartners-Google Yandex Googlebot proximic AmazonAdBot Amazon-Advertising-ad-standards-bot/1.0 APS-ad-standards-bot/1.0 *


* GPTBot ChatGPT-User ClaudeBot Claude-Web anthropic-ai PerplexityBot TurnitinBot cohere-ai OAI-SearchBot http://quillbot.com Google-Extended


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


GoogleBot YandexBot YandexImages bingbot Baiduspider *


Googlebot
Googlebot-Image
Bingbot
AdIdxBot
BingPreview
DuckDuckBot
Baiduspider
Yandex
Applebot
msnbot
Naverbot
Slurp
teoma
facebookexternalhit
Twitterbot
LinkedInBot
Pinterest
WhatsApp
Pubmatic Crawler Bot
*


* Googlebot Bingbot Applebot DuckDuckBot GPTBot ChatGPT-User ClaudeBot Claude-User Google-Extended CCBot Meta-ExternalAgent Applebot-Extended PerplexityBot YouBot Twitterbot


* KDDI-Googlebot-Mobile bingbot dotbot Bytespider


*
DMCAAgent
CopyrightCrawler
AntiPiracyBot
008
Slurp
goo-crawler
RakutenBot
Teoma
Baiduspider
Sogou Spider
360Spider
YisouSpider
ia_archiver
MusoBot
Link-BusterBot


* AhrefsBot MJ12bot SemrushBot-SA SemrushBot-BA SemrushBot-SI SemrushBot-SWA SemrushBot-CT SemrushBot-BM


* Yandex AdsBot-Google facebookexternalhit Twitterbot


MJ12bot SemrushBot SemrushBot-SA serpstatbot dotbot BLEXBot barkrowler RSiteAuditor CLASSLA-web ImagesiftBot PetalBot Paqlebot Baiduspider * Bingbot MSNBot BingPreview


Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot[at]gmail[dot]com)

Crescent Internet ToolPak HTTP OLE Control v.1.0


* CCBot GPTBot Google-Extended PerplexityBot ias_crawler


Xombot AhrefsBot Zing-BottaBot Zing-BottaBot/2.0 GoScraper GPTBot Google-Extended dotbot MJ12bot BLEXBot trendictionbot Barkrowler Turnitin MojeekBot BingBot PerplexityBot ChatGPTBot GeminiBot *


Claude-User Claude-Web Claude-SearchBot ChatGPT-User OAI-SearchBot PerplexityBot Perplexity-User GPTBot ClaudeBot CCBot Google-Extended Bytespider meta-externalagent bingbot ia_archiver *


* Googlebot Googlebot-News Bingbot Slurp Twitterbot facebookexternalhit Applebot baiduspider


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent Mail.Ru Baiduspider MSNBot Bingbot Slurp Yandex


* semetrical GPTBot OAI-SearchBot Applebot-Extended PerplexityBot Perplexity-ai CCBot YouBot ClaudeBot anthropic-ai Claude-Web grapeshot AmazonAdBot ozone Meta-ExternalAgent


* Googlebot Googlebot-image AdsBot-Google Twitterbot


GPTBot ClaudeBot Amazonbot Applebot Applebot-Extended CCbot Bytespider PetalBot FacebookBot MegaIndex.ru OAI-SearchBot ChatGPT-User Claude-User Claude-SearchBot Amzn-Searchbot Amzn-User AmazonAdBot PerplexityBot *


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* 8LEGS AhrefsBot AspiegelBot BLEXBot Barkrowler DotBot MJ12bot MauiBot Nimbostratus-Bot PetalBot SemrushBot SeznamBot Sogou serpstatbot trendiction TextBulkerBot Amazonbot Bytespider CCBot cohere-ai FacebookBot Omgilibot


YandexBot
dotbot
SemrushBot
SemrushBot-SA
VegeBot
Vegi bot
BaiduSpider
BLEXBot
panscient.com
AhrefsBot
PiplBot
AlphaSeoBot
PetalBot
facebookexternalhit
*


* Claude-User FacebookBot meta-externalfetcher CCBot cohere-ai Claude-Web PiplBot GPTBot MuckRack PerplexityBot magpie-crawler Google-Extended OAI-SearchBot ClaudeBot omgilibot Gensparkbot ICC-Crawler MJ12bot Timpibot Webzio Perplexity-User cohere-training-data-crawler Scrapy Diffbot omgili Applebot-Extended Meta-ExternalAgent Claude-SearchBot Bytespider AmazonBot ChatGPT-User Perplexity-ai Webzio-Extended Webzio-extended anthropic-ai


*
AhrefsBot
Aliyun
AliyunSecBot
AliyunSecBot/Aliyun
Amazonbot
Amzn-SearchBot
Baiduspider
barkrowler
BLEXBot
Brightbot 1.0
Bytespider
CCBot
ClaudeBot
Claude-Web
Cliqzbot
coccocbot
DataForSeoBot
dotbot
GPTBot
ImagesiftBot
Linguee
megaindex
meta-externalagent
Nimbostratus
oBot
PerplexityBot
PetalBot
proximic
Seekport Crawler
SemrushBot
serpstatbot
SleepBot
SMTBot
Sogou web spider
SurdotlyBot
TikTokSpider
Turnitin Robot
YouBot


* FacebookExternalHit Twitterbot Pinterestbot LinkedInBot TikTokSpider Storebot-Google Google-InspectionTool AmazonAdBot Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended Amazonbot meta-webindexer meta-externalagent meta-externalfetcher ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User cohere-training-data-crawler cohere-ai CCBot PanguBot PetalBot Bytespider Diffbot DuckAssistBot MistralAI-User Timpibot Webzio Webzio-Extended archive.org_bot ia_archiver ia_archiver-web.archive.org heritrix


*
Anthropic-ai
ChatGPT-User
ClaudeBot
Claude-Web
Cohere-ai
GPTBot
GPT
ImagesiftBot
PerplexityBot
YouBot
AndiBot
MindsDBBot
MetaphorSystemsBot
PhindBot
WolframAlphaBot
oai-searchbot
You.com
Applebot-Extended
Google-Extended
Seekr
MistralAI-User
ChatDK-Bot
SBIntuitionsBot
DuckAssistBot
quillbot
MojeekBot
Yandex
YaDirectFetcher
Baiduspider
Sosospider
YodaoBot
Yeti
Sogou
Bytespider
Mail.Ru
SeznamBot
coccocbot-web
AhrefsBot
AhrefsSiteAudit
SemrushBot
MajesticBot
MozBot
UbersuggestBot
serpstatbot
BrightEdgeBot
SiteAuditBot
cognitiveSEO
DataForSeoBot
RankActiveBot
SpyFuBot
SEOlyticsCrawler
RankurBot
RyteBot
Monsidobot
Cocolyzebot
siteimprove.com/bots
MOSBot
BLEXBot
DotBot
MJ12bot
Diffbot
Diffbot/1.0
Diffbot/2.0
ZoominfoBot
intelx.io_bot
SupabaseEdgeScraper
SaaSBrowserBot
LinkupBot
domainsbot
URLChecker
URLChecker/1.0
URLChecker/2.0
URLCheckerBot
fynd.bot
MatchMyVC-bot
affiliate-wiki-bot
ScannerBot
WebAuditScannerBot
Vladhog
WellKnownBot
d-bot-health-check
uptimerobot
Thinkbot
Newslitbot
GnowitNewsbot
Poggio-Citations
ScourRSSBot
smry.ai
rss-is-dead.lol
rss-is-dead.lol web bot
Paqlebot
SeekportBot
Cubot
yacybot
Dragonbot
Dubbotbot
EzoicBot-Nicheiq
SurdotlyBot
TSMbot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* GPTBot OAI-SearchBot ChatGPT-User ClaudeBot anthropic-ai Claude-Web PerplexityBot Google-Extended CCBot Bytespider FacebookBot Meta-ExternalAgent Applebot-Extended Amazonbot cohere-ai DuckAssistBot


Crescent Internet ToolPak HTTP OLE Control v.1.0


Baiduspider Baiduspider-video Baiduspider-image *


*
AhrefsBot
AI2Bot
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
Arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
coccocbot-web
cohere-ai
cohere-training-data-crawler
Crawlspace
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
GPTBot
https://hada.news
ICC-Crawler
ImagesiftBot
https://www.imediaethics.org
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
magpie-crawler
Meta-ExternalAgent
Meta-ExternalFetcher
Mojeek
MojeekBot
MoodleBot
Netvibes
NewsNow
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
PetalBot
Scrapy
Seekr
SeznamBot
Timpibot
Turnitin
TurnitinBot
VelenPublicWebCrawler
Webzio-Extended
yacy
yacybot
Yandex
YandexAdditional
YandexAdditionalBot
YouBot
Perplexity-User
Claude-User
Claude-SearchBot
Webzio
Gemini-Deep-Research
MistralAI-User
SemrushBot-OCOB


Mozilla/5.0 (compatible; Qwantify/; +https://www.qwant.com/)


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


Mediapartners-Google YisouSpider Leikibot ShopWiki NPBot psbot HenryTheMiragoRobot gigabot appie Meta-ExternalAgent AhrefsBot *


Mediapartners-Google Adsbot-Google Googlebot-Image *


* Twitterbot Yeti Baiduspider Ezooms bingbot


ADmantX
MauiBot
CCBot
VoluumDSP
MediaPartners-Google
Cliqzbot
BLEXBot
Applebot
AhrefsBot
MegaIndex.ru
megaindex.com
Bytespider
serpstatbot
AspiegelBot
Baiduspider
Sosospider
sogou spider
Sogou web spider
360spider
YisouSpider
YodaoBot
MJ12bot
grapeshot
MTRobot
Barkrowler
SemrushBot
VelenPublicWebCrawler
Mail.RU_Bot
DataForSeoBot
Amazonbot
YandexBot
Leikibot
GPTBot
AwarioBot
meta-externalagent
PlurkBot
ClaudeBot
Claude-SearchBot
DotBot
*


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* adsbot-google Nutch AhrefsBot AhrefsSiteAudit MJ12bot Pinterest


OpenAI CCBot ChatGPT ChatGPT-User GPTBot Google-Extended anthropic-ai cohere-ai ia_archiver omgili omgilibot MJ12bot PiplBot Claude-Web ClaudeBot FacebookBot Amazonbot Bytespider magpie-crawler PerplexityBot AhrefsBot Bytedance Diffbot TurnitinBot Quora-Bot Meta-ExternalAgent Applebot-Extended Google-CloudVertexBot DeepSeekBot xAI Grok *


* AhrefsBot MJ12bot SemrushBot-SA SemrushBot-BA SemrushBot-SI SemrushBot-SWA SemrushBot-CT SemrushBot-BM


* Googlebot Mediapartners-Google Adsbot-Google Bingbot msnbot Slurp Twitterbot popIn_Agent


UbiCrawler
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper
Baiduspider
baiduspider
*


* Yandex AdsBot-Google facebookexternalhit Twitterbot


Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent


* MSNBot Bingbot YandexBot Baiduspider Sogou SemrushBot AhrefsBot BLEXBot MJ12bot 008 sistrix ia_archiver proximic SiteExplorer SemrushBot-SA SemrushBot-BA SemrushBot-SI SemrushBot-SWA SemrushBot-CT SemrushBot-BM SplitSignalBot


*
expo9
Orthogaffe
UbiCrawler
DOC
Zao
Zealbot
MSIECrawler
SiteSnagger
WebStripper
Fetch
OfflineExplorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Xenu
larbin
libwww
ZyBORG
DownloadNinja
wget
grub-client
WebCopier
Python-urllib
URL_Spider_Pro
EmailCollector
EmailSiphon
WebBandit
EmailWolf
ExtractorPro
CopyRightCheck
Alexibot
Zeus
ZeusLinkScout
EroCrawler
LinkScan
WebCapture 2.0
EtaoSpider
Black Hole
Yahoo! Slurp China
Yandex
YandexTurbo
YandexBot
msnbot
Slurp
Nutch
008
CherryPicker


*
YandexVideoParser
008
BLEXBot
AhrefsBot
Vagabondo
SEOkicks-Robot
ia_archiver
archive.org_bot
special_archiver
MJ12bot
heritrix
netEstate NE Crawler
sistrix
WBSearchBot
QuerySeekerSpider
proximic
SiteExplorer
SemrushBot
SemrushBot-SA
Mail.Ru
Baiduspider
MSNBot
Bingbot
Slurp
Yandex


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot Google-Extended GPTBot meta-externalagent


Baiduspider Yandex Mail.Ru AhrefsBot coccoc SemrushBot SemrushBot-SA Scrapy Riddler SemanticScholarBot *


AddSearchBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
amazon-kendra
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Amzn-User
Andibot
Anomura
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-Extended
Aranet-SearchBot
atlassian-bot
Awario
AzureAI-SearchBot
bedrockbot
bigsur.ai
Bravebot
Brightbot 1.0
BuddyBot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Crawlspace
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
ExaBot
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
GPTBot
iAskBot
iaskspider
iaskspider/2.0
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
img2dataset
ISSCyberRiskCrawler
kagi-fetcher
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
LAIONDownloader
LCC
LinerBot
Linguee Bot
LinkupBot
Manus-User
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NotebookLM
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poggio-Citations
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot


* BacklinkCrawler Backlink-Ceck backlink-check MJ12bot AhrefsBot SemrushBot


*
A6-Indexer
AddSearchBot
aiHitBot
Andibot
anthropic-aiBytespider
AspiegelBot
archive.org_bot
Barkrowler
bigsur.ai
BLP_bbot
Brightbot 1.0
Buck
ChatGLM-Spider
coccocbot-web
Cotoyogi
crawler4j
Crawlspace
Datenbank Crawler
cohere-training-data-crawler
cohere-ai
omgili
omgilibot
Omgilibot
Applebot-Extended
Bytespider
CCBot
ChatGPT
Diffbot
EasouSpider
Echobot Bot
EchoboxBot
ecoResearch
FirecrawlAgent
Genieo
https://hada.news
ia_archiver
ia_archiver-web.archive.org
iaskspider/2.0
ICC-Crawler
img2dataset
JenkersBot
Jetslide
Kangaroo Bot
KunatoCrawler
LivelapBot
magpie-crawler
MauiBot
MAZBot
MistralAI-User
MistralAI-User/1.0
Mojeek
MojeekBot
MyCentralAIScraperBot
news-please
NewsNow
NovaAct
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
PetalBot
PanguBot
Poseidon Research Crawler
Primalbot
QuillBot
quillbot.com
R6_CommentReader
SBIntuitionsBot
Scrapy
SeekportBot
Seekr
SeekrBot
SeznamBot
SeznamHomepageCrawler
Sidetrade indexer bot
Spinn3r
TaraGroup Intelligent Bot
Thinkbot
Timpibot
VelenPublicWebCrawler
Verity
Verity/1.1
ViennaTinyBot
WebVac
Webzio-Extended
WebZip
WRTNBot
yacy
yacybot
YouBot
ankiro.com
CheckMarkNetwork
dataprovider
legistorm.com
Magus Bot
Netvibes
nlcrawler
NLCrawler
Oracle Secure Enterprise Search
panscient.com


Baiduspider baiduspider Baiduspider+ Baiduspider-video Baiduspider-image asterias Spiderbot/Nutch-1.7 *


Crescent Internet ToolPak HTTP OLE Control v.1.0


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


Mediapartners-Google MJ12bot Ezooms Riddler * SeznamBot AI


* Googlebot Adsbot-Google Mediapartners-Google


* Claude-SearchBot OAI-SearchBot GPTBot ClaudeBot SemrushBot AhrefsBot MJ12bot DotBot Amazonbot Bytespider TikTokSpider ShapBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* PetalBot BLEXBot Neevabot DataForSeoBot Yandex AhrefsBot Sogou Paqlebot serpstatbot TurnitinBot Buck SemrushBot trendictionbot Amazonbot anthropic-ai Bytespider CCBot ChatGPT-User Claude-Web Google-Extended GPTBot


GPTBot ChatGPT-User MSNbot FacebookBot ImagesiftBot AhrefsBot MJ12bot weborama-fetcher


*
Googlebot
Googlebot-News
Googlebot-Image
Googlebot-Video
Googlebot-Mobile
Google-Extended
Twitterbot
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper


* Aolbot-News GPTbot anthropic-ai CCBot OmigiliBot Bytespider Diffbot ClaudeBot


* ia_archiver urlscan.io Swiftbot AdsBot-Google Mediapartners-Google


* OAI-SearchBot ChatGPT-User PerplexityBot ClaudeBot Googlebot Bingbot


*
Baiduspider
Sogou Spider
360Spider
Yandex
Bytespider
PetalBot
MJ12bot
AhrefsBot
curl
wget
python-requests
HTTrack


* AdsBot-Google Googlebot-Image Mediapartners-Google trovitBot kalooga MJ12bot proximic grapeshot GrapeshotCrawler WeSEE WeSEE:Search admantx Exabot SkimBot Ezooms AhrefsBot Mail.RU Mail.RU_Bot CrystalSemanticsBot URLAppendBot


* Googlebot Bingbot WhatsApp Slackbot AppEngine-Google Mediapartners-Google GPTBot PerplexityBot ChatGPT-User ClaudeBot anthropic-ai cohere-ai DuckAssistBot Meta-ExternalAgent OAI-SearchBot Quora-Bot PetalBot DataForSeoBot


*
YandexVideoParser
008
BLEXBot
AhrefsBot
Vagabondo
SEOkicks-Robot
ia_archiver
archive.org_bot
special_archiver
MJ12bot
heritrix
netEstate NE Crawler
sistrix
WBSearchBot
QuerySeekerSpider
proximic
SiteExplorer
SemrushBot
SemrushBot-SA
Mail.Ru
Baiduspider
MSNBot
Bingbot
Slurp
Yandex


anthropic-ai
AmazonBot
Claude
Claude-Web
claudeweb
Amzn-SearchBot
Amzn-User
AndiBot
cohere-ai
ClaudeBot
ChatGPT-User
Googlebot-News
GPTBot
CCBot
Google-Extended
DataForSeoBot
Diffbot
Diffbot-user
Kagibot
FacebookBot
magpie-crawler
NewsNow
news-please
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Scrapy
TurnitinBot
Applebot-Extended
Meta-ExternalAgent
meta-externalagent
OAI-SearchBot
Bytespider
Claude-User
Claude-SearchBot
AI2Bot
AhrefsBot
Ai2Bot-Dolma
Applebot
Arquivo-web-crawler
AspiegelBot
AwarioRssBot
AwarioSmartBot
Barkrowler
coccocbot-web
cohere-training-data-crawler
crawler4j
Crawlspace
dotbot
DuckAssistBot
FriendlyCrawler
Google-CloudVertexBot
https://hada.news
https://www.imediaethics.org
iaskspider/2.0
ICC-Crawler
icc-crawler
ICC Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
LivelapBot
MauiBot
Meltwater
Meta-ExternalFetcher
MJ12bot
Mojeek
MojeekBot
MoodleBot
NeticleBot
Netvibes
PanguBot
PetalBot
SeekportBot
Seekr
SemrushBot
SeznamBot
Sidetrade Indexer Bot
Timpibot
Turnitin
VelenPublicWebCrawler
Webzio-Extended
yacy
yacybot
Yandex
YandexAdditional
YandexAdditionalBot
YandexBot
YandexImages
YandexNews
Baiduspider
Baiduspider-image
BaiduSpider-render
Baiduspider-video
Sogou Web Spider
yisouspider


* AdsBot-Google Googlebot Googlebot-Image


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


*
noxtrumbot
msnbot
Slurp
MSIECrawler
WebCopier
HTTrack
Microsoft.URL.Control
libwww
Orthogaffe
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
SiteSnagger
WebStripper
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
Xenu
larbin
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper
Googlebot


Twitterbot * GPTBot ChatGPT-User CCBot anthropic-ai PerplexityBot Amazonbot Applebot-Extended Bytespider ClaudeBot Claude-Web cohere-ai Diffbot Google-Extended Meta-ExternalFetcher YouBot DeepSeekBot MistralAI-User MistralAI-User/1.0 PetalBot Timpibot YandexAdditional YandexAdditionalBot YandexBot YandexRenderBot Perplexity-User coccocbot-web OAI-SearchBot


Amazonbot GPTBot AhrefsBot Applebot DataForSeoBot ByteDance Bytespider ClaudeBot *


* 008 PetalBot DataForSeoBot AwarioRssBot AwarioSmartBot Sogou Baiduspider YisouSpider AhrefsBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


PetalBot GPTBot CCBot anthropic-ai Claude-Web Bytespider *


* Mediapartners-Google SemrushBot proximic Twitterbot


AmazonAdBot Mediapartners-Google grapeshot YandexBot SemrushBot HybridBot MegaIndex dotbot BLEXBot MJ12bot proximic *


AndiBot Applebot Applebot-Extended bingbot Bytespider CCBot ChatGPT-User Claude-SearchBot Claude-User ClaudeBot DuckAssistBot ExaBot FirecrawlAgent Google-Agent Google-Extended Googlebot GPTBot meta-externalagent meta-externalfetcher meta-webindexer Perplexity-User PerplexityBot PhindBot YouBot *


GPTBot Amazonbot ClaudeBot Bytespider CCBot PerplexityBot *


* GPTBot ChatGPT-User anthropic-ai Claude-Web


* GPTBot ChatGPT-User ClaudeBot PerplexityBot Google-Extended


ia_archiver Baiduspider BaiduImagespider yodaobot Yeti NaverBot Googlebot *


* Nuclei WikiDo Riddler PetalBot Zoominfobot Go-http-client Node/simplecrawler CazoodleBot dotbot/1.0 Gigabot Barkrowler BLEXBot magpie-crawler


claritybot GPTBot Google-Extended CCBot ChatGPT-User anthropic-ai Omgilibot Omgili FacebookBot Bytespider *


Mediapartners-Google
Googlebot
Googlebot-Image
Googlebot-Mobile
Googlebot-News
Googlebot-Video
Adsbot-Google
Twitterbot
Applebot
Bingbot
SiteAuditBot
Publication-Access-for-Facebook
facebookexternalhit
Flipboard
FlipboardProxy
upday
AhrefsBot
AhrefsSiteAudit
5emeRue
5erue
adequat
adequat-systems
AmiSoftware
AwarioRssBot
AwarioSmartBot
Argus
Ask n read
asknread.com
Augure
auramundi
Bloodhound
Cision
coexel
ConveraCrawler
Corporama
cydralspider
Digimind
Download Ninja
downloadexpress
EDD
ellisphere
eureka
Europresse
Explore
Factiva
Fasterfox
Fetch
gammaSpider
grub-client
HTTrack
ia_archiver
ia_archiver-web.archive.org
indexer
infoseek
Jetbot
k2spider
Kantar
kbcrawl
Knowings
larbin
leadbox
libwww
linkfluence
linko
manageo
mediacompil
Meltwater
mention
Moreover
MSIECrawler
mytwip
newscan-online
NewsNow
Newzbin
NPBot
ObjectsSearch
Offline Explorer
opinion-tracker
Pimptrain
proxem
QuepasaCreep
Qwam content intelligence
Raven
readability.com
scoop.it
score3
Sindup
sitecheck.internetseer.com
SiteSnagger
spotter
Synthesio
Talkwater
Teleport
TeleportPro
trendeo
trendybuzz
TunitinBot
TurnitinBot
up2news
vecteurplus


*
Yandex
Googlebot
Java
NjuiceBot
Gigabot
Scrapy
Baiduspider
SeznamBot
crawler
JS-Kit
HybridBot
Voyager
PostRank
DomainCrawler
SemrushBot
MegaIndex.ru
ltx71
SurveyBot
AhrefsBot
Exabot
Aport
CCBot
DotBot
ia_archiver
larbin
Butterfly
libwww
bingbot
Wget
SWeb
LinkExchanger
Soup
GrapeshotCrawler
WordPress
DnyzBot
spbot
DeuSu
MLBot
InternetSeer
BUbiNG
FairShare
Yeti
Birubot
YottosBot
Linguee
Ezooms
lwp-trivial
Purebot
kmSearchBot
SiteBot
CamontSpider
ptd-crawler
HTTrack
suggybot
ttCrawler
Nutch
msnbot
msnbot-media
Slurp
Zeus
Abonti
aggregator
asterias
BDCbot
BLEXBot
BuiltBotTough
Bullseye
BunnySlippers
Cegbfeieh
CheeseBot
CherryPicker
coccoc
CopyRightCheck
cosmos
Crescent
discobot
DittoSpyder
Download Ninja
EasouSpider
EmailCollector
EmailSiphon
EmailWolf
EroCrawler
ExtractorPro
Fasterfox
FeedBooster
Foobot
Genieo
Harvest
hloader
httplib
humanlinks
ieautodiscovery
Incutio
InfoNaviRobot
IstellaBot
JamesBOT
JennyBot
k2spider


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* MJ12bot J12bot coccocbot Googlebot Yeti


Amazonbot Anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web Cohere-ai DataForSeoBot Google-Extended GPTBot ImagesiftBot Magpie-crawler Omgili Omgilibot Peer39_crawler Peer39_crawler/1.0 PerplexityBot YouBot Googlebot Googlebot-Image Googlebot-News Googlebot-Video AdSense AdsBot-Google AdsBot Google-InspectionTool APIs-Google Feedfetcher-Google Google-NotebookLM facebookexternalhit Facebot meta-externalagent Facebook Meta *


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


* Yandex MJ12bot AhrefsBot grapeshot VelenPublicWebCrawler CriteoBot/0.1 PetalBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


AI2Bot Amazonbot anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User Claude-Web ClaudeBot cohere-ai DataForSeoBot Diffbot DuckAssistBot FacebookBot Google-Extended GPTBot ImagesiftBot Magpie-crawler Omgili Omgilibot Peer39_crawler Peer39_crawler/1.0 PerplexityBot PetalBot Timpibot YouBot *


* AmazonAdBot Google-Extended BLP_bbot BLP_bbot/0.1 CCBot anthropic-ai cohere-ai ia_archiver omgili omgilibot MJ12bot PiplBot Bytespider ClaudeBot Claude-Web FacebookBot magpie-crawler NewsNow news-please PerplexityBot Perplexity-ai Meta-ExternalAgent Meta-ExternalFetcher Applebot-Extended


anthropic-ai GPTBot ClaudeBot Claude-web CCBot Google-Extended Amazonbot Applebot Bingbot ChatGPT-User Bytespider PerplexityBot Sogou CoveoEnterpriseSearch * AhrefsBot YandexBot MegaIndex.ru SemrushBot Qwantify/Bleriot DotBot MJ12bot SEOkicks


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent Googlebot-Image


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot Google-Extended GPTBot meta-externalagent


Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)


*
008
BLEXBot
AhrefsBot
Vagabondo
SEOkicks-Robot
ia_archiver
archive.org_bot
special_archiver
MJ12bot
heritrix
netEstate NE Crawler
sistrix
WBSearchBot
QuerySeekerSpider
proximic
SiteExplorer
SemrushBot
SemrushBot-SA
Baiduspider
MSNBot
Bingbot


GPTBot Google-Extended ClaudeBot Bytespider CCBot Applebot-Extended FacebookBot omgili *


*
DMCAAgent
CopyrightCrawler
AntiPiracyBot
008
Slurp
goo-crawler
RakutenBot
Teoma
Baiduspider
Sogou Spider
360Spider
YisouSpider
ia_archiver
MusoBot
Link-BusterBot


* AhrefsBot dotbot MJ12bot SemrushBot SemrushBot-SA


anthropic-ai
AmazonBot
Claude
Claude-Web
claudeweb
Amzn-SearchBot
Amzn-User
AndiBot
cohere-ai
ClaudeBot
ChatGPT-User
Googlebot-News
GPTBot
CCBot
Google-Extended
DataForSeoBot
Diffbot
Diffbot-user
Kagibot
FacebookBot
magpie-crawler
NewsNow
news-please
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Scrapy
TurnitinBot
Applebot-Extended
Meta-ExternalAgent
meta-externalagent
OAI-SearchBot
Bytespider
Claude-User
Claude-SearchBot
AI2Bot
AhrefsBot
Ai2Bot-Dolma
Applebot
Arquivo-web-crawler
AspiegelBot
AwarioRssBot
AwarioSmartBot
Barkrowler
coccocbot-web
cohere-training-data-crawler
crawler4j
Crawlspace
dotbot
DuckAssistBot
FriendlyCrawler
Google-CloudVertexBot
https://hada.news
https://www.imediaethics.org
iaskspider/2.0
ICC-Crawler
icc-crawler
ICC Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
LivelapBot
MauiBot
Meltwater
Meta-ExternalFetcher
MJ12bot
Mojeek
MojeekBot
MoodleBot
NeticleBot
Netvibes
PanguBot
PetalBot
SeekportBot
Seekr
SemrushBot
SeznamBot
Sidetrade Indexer Bot
Timpibot
Turnitin
VelenPublicWebCrawler
Webzio-Extended
yacy
yacybot
Yandex
YandexAdditional
YandexAdditionalBot
YandexBot
YandexImages
YandexNews
Baiduspider
Baiduspider-image
BaiduSpider-render
Baiduspider-video
Sogou Web Spider
yisouspider


Crescent Internet ToolPak HTTP OLE Control v.1.0


DotBot AhrefsBot SemrushBot MJ12bot IASBot *


* GPTBot ChatGPT-User PerplexityBot ClaudeBot Google-Extended Amazonbot Applebot-Extended CCBot


*
AddSearchBot
AI2Bot
Ai2Bot-Dolma
aiHitBot
amazon-kendra-
Amazonbot
Andibot
Anomura
anthropic-ai
Applebot
Applebot-Extended
Awario
Baiduspider
bedrockbot
bigsur.ai
Bravebot
Brightbot 1.0
Bytespider
CCBot
GPTBot
ChatGPT Agent
ChatGPT-User
ClaudeBot
Claude-Web
Claude-SearchBot
Claude-User
Cloudflare-AutoRAG
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
EvilAISample
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
Google-Firebase
Google-NotebookLM
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
iaskspider/2.0
ia_archiver
IbouBot
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
LinerBot
magpie-crawler
meta-webindexer
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
ShapBot
Sidetrade indexer bot
TerraCotta
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YaK


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* BLEXBot meta-externalagent MJ12bot SemrushBot serpstatbot AhrefsBot Baiduspider bingbot


* Googlebot AdsBot-Google Googlebot-Image


Googlebot Bingbot Google-Extended Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot GPTBot MJ12bot DotBot PetalBot SemrushBot AhrefsBot DataForSeoBot Crawler4j Scrapy *


MJ12bot YandexBot DataForSeoBot Twitterbot facebookexternalhit bingbot msnbot GPTBot Applebot *


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
Meltwater
Cision
Talkwater
Jetbot
kbcrawl
Newzbin
Qwam content intelligence
flipboard
Youmag
Synthesio
trendybuzz
scoop.it
linkfluence
grub-client
ia_archiver-web.archive.org
k2spider
libwww
wget
5erue
adequat
adequat-systems
coexel
leadbox
mention
mytwip
opinion-tracker
proxem
score3
vecteurplus
verticalsearch
vsw
winello
Fetch
infoseek
MSIECrawler
Offline Explorer
sitecheck.internetseer.com
Teleport
TeleportPro
WebCopier
WebStripper
Zealbot
asknread.com
omgilibot
omgili
CCBot
Google-Extended
PerplexityBot
FacebookBot
anthropic-ai
Claude-Web
ClaudeBot
Applebot-Extended
Webzio-Extended
Meta-ExternalFetcher
Amazonbot
Timpibot
AI2Bot
Applebot
cohere-training-data-crawler
DuckAssistBot
Kangaroo Bot
Meta-ExternalAgent
PanguBot
5emeRue
ACQUIRE MEDIA
ACTIV Financial (CME Group)
AlphaSense
AmiSoftware
archive.org_bot
Archive-It
ArgClrInt
Ask n read
Augure
auramundi
AwarioRssBot
AwarioSmartBot
Barchart.com
BattleFin
Bernin IT
Blackboard Safeassign
BLP_bbot
bluematrix
Brandwatch
Briefcase.news
Buck
Bytespider
CCBo
CikisiBot
Coexel
cohere-ai
Comtex News Network
ConveraCrawler
Copyright Licensing Agency
Corporama
D&B Hoovers
Data Expression
Data Observer
Dataminr


* Googlebot Googlebot-Image Googlebot-Mobile Googlebot-Video Mediapartners-Google


*
archive-it.org
archive.org_bot
ArchiveBot
Arquivo-web-crawler
Authory
babel
bl.uk_ldfc_bot
bl.uk_ldfc_renderbot
bne.es_bot
bnf.fr_bot
ecoresearch
ecoresearchCrawler
Europarchive.org
kb.dk_bot
Kulturarw3
LivelapBot
mediacloud
mirrorweb
MSIECrawler
nationalarchives.gov.uk
netarkivet.dk
netEstate NE Crawler
Nicecrawler
nl-israel_iaharvester2026
nlnbot
nlncrawler
PageFreezer
SafeDNS
SmarshBot
special_archiver
tkl.iis.u-tokyo.ac.jp
TurnitinBot
webarchiv.cz
WordPress.com mShots
XY-Archive-Compliance-Archiver
XY-Archive-Compliance-Crawler
AhrefsBot
AI2Bot
Ai2Bot-Dolma
Amazonbot
amazon-kendra-
anthropic-ai
Applebot-Extended
bedrockbot
Bytespider
CloudflareBrowserRenderingCrawler
CCBot
ChatGLM-Spider
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
cohere-ai
Cotoyogi
DeepSeekBot
Diffbot
DuckAssistBot
EchoboxBot
FacebookBot
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
Grok
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
KunatoCrawler
LinerBot
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MistralAI-User
OAI-Operator
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
Perplexity-User
PetalBot
QualifiedBot
Scrapy
Seekr
Sidetrade indexer bot
TaraGroup Intelligent Bot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended


*
Baiduspider
Yandex
DataForSeoBot
Screaming Frog SEO Spider
SemrushBot
Meta-ExternalAgent
Applebot
Amazonbot
BLEXBot
PetalBot
DotBot
Bytespider
ClaudeBot
GPTBot
ChatGPT-User
OAI-SearchBot
AmazonAdBot


claritybot GPTBot Google-Extended CCBot ChatGPT-User anthropic-ai Omgilibot Omgili FacebookBot Bytespider *


* AmazonAdBot bytespider mistralai cohere ai2bot youbot omgili diffbot kangaroo img2dataset amazonbot amazon-qbusiness meta-externalagent meta-webindexer


* BLEXBot MJ12bot SemrushBot PetalBot CriteoBot proximic grapeshot ia_archiver


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent archive.org_bot BoardReader PerplexityBot FacebookBot OmgiliBot anthropic-ai cohere-ai


QuerySeekerSpider ( http://queryseeker.com/bot.html )


*
Googlebot
AhrefsBot
noxtrumbot
msnbot
Slurp
MSIECrawler
WebCopier
HTTrack
Microsoft.URL.Control
libwww
Orthogaffe
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
SiteSnagger
WebStripper
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
Xenu
larbin
ZyBORG
Download Ninja
wget
grub-client
k2spider
NPBot
WebReaper


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent Googlebot Bingbot Slurp Teoma ia_archiver twiceler MSNBot Scrubby Robozilla Gigabot googlebot-image googlebot-mobile yahoo-mmcrawler yahoo-blogs/v3.9 psbot OpenAIOpenAI Applebot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent ChatGPT-User SemrushBot AhrefsBot PetalBot BLEXBot Linguee MJ12bot DotBot


Googlebot Bingbot Applebot DuckDuckBot OAI-SearchBot ChatGPT-User Claude-SearchBot Claude-User PerplexityBot Perplexity-User DuckAssistBot GPTBot ClaudeBot Google-Extended Applebot-Extended CCBot Amazonbot cohere-ai Bytespider meta-externalagent ImageSiftBot Diffbot Omgilibot Omgili PetalBot MJ12bot SemrushBot AhrefsBot Claude-Web anthropic-ai *


008
AddSearchBot
AI2Bot
Ai2Bot-Dolma
Ai2Bot-DeepResearchEval
Amazonbot
AmazonBuyForMe
Anchor Browser
Anomura
anthropic-ai
atlassian-bot
AwarioSmartBot
Applebot
Applebot-extended
bigsur.ai
Brandwatch
Bravebot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank-Crawler
DDM-DCipher
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
FacebookBot
Factset_spyderbot
GPTBot
HTTrack
iAsk
iaskspider
ICC-Crawler
ImageSiftBot
imageSpider
Kangaroo-Bot
KlaviyoAIBot
Kunato
laion-huggingface-processor
LinerBot
LCC
LinkupBot
Manus-User
meta-externalagent
meta-externalfetcher
meta-webindexer
mistral.ai
MistralAI-User
netEstate Imprint Crawler
NovaAct
Novellum
Nutch
OAI-SearchBot
Offline Explorer
omgili
omgilibot
PanguBot
PeopleInc-DCipher-Scraper/1.1.0
Perplexity-User
PerplexityBot
PetalBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
Scrapy
SeekrBot
SemrushBot-OCOB
SemrushBotSwa
ShapBot
Spider
TavilyBot
Timpibot
TwinAgent
Velen Crawler
VelenPublicWebCrawler
Webzio-Extended
Wrtn
YisouSpider
YouBot
ZanistaBot
*
AdsBot-Google


SleepBot ClaudeBot GPTBot Amazonbot meta-externalagent Baiduspider AhrefsBot *


Mediapartners-Google SeekportBot Yandex GPTBot GPTCrawler ChatGPT-User OAI-SearchBot ClaudeBot anthropic-ai Google-Extended CCBot PerplexityBot Perplexity-User Bytespider Bytespider-webpage Amazonbot *


*
W3C-checklink
W3C_Validator
W3C_CSS_Validator
Baiduspider
archive.org_bot
ia_archiver
FavIconizer
AhrefsBot
A6-Indexer/1.0
butterfly
Exabot
grapeshot
Mail.Ru
nutch
pingdom
proximic
Python
SearchmetricsBot
SeznamBot
Sogou web spider
sogou spider
Sogou
YoudaoBot
Sosospider
spbot
Wget
yie8
MJ12bot
360Spider
TurnitinBot
BLEXBot
hivaBot
NerdyBot
updown_tester
AndroidDownloadManager
voltron
Xenu's Link Sleuth 1.1c
SemrushBot
SemrushBot-SA
oBot
Riddler
Mail.RU_Bot


* MJ12bot AhrefsBot BLEXBot Yandex baiduspider SemrushBot BingBot


BLEXBot MJ12bot CriteoBot/0.1 SemrushBot PetalBot proximic grapeshot ia_archiver AhrefsBot *


* GPTBot ChatGPT-User CCBot anthropic-ai ClaudeBot Google-Extended PerplexityBot Bytespider SemrushBot AhrefsBot DotBot


* Googlebot AdsBot-Google AdsBot-Google-Mobile Googlebot-Image


* Yeti Googlebot ZumBot SemrushBot YandexBot


* Googlebot Googlebot-News Googlebot-Image Googlebot-Video Storebot-Google Google-InspectionTool GoogleOther Google-Extended AdsBot-Google-Mobile AdsBot-Google Mediapartners-Google Bingbot MSNBot MSNBot-Media AdIdxBot BingPreview AmazonAdBot Slurp DuckDuckBot Applebot FacebookExternalHit Twitterbot LinkedInBot


AmazonAdBot * seekbot Exabot Slurp Cliqzbot AI2Bot ChatGLM ChatGLM-spider cohere-training-data-crawler Diffbot OmigiliBot PetalBot YouBot


* Nuclei WikiDo Riddler PetalBot Zoominfobot Go-http-client Node/simplecrawler CazoodleBot dotbot/1.0 Gigabot Barkrowler BLEXBot magpie-crawler


Amazonbot Anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web Cohere-ai DataForSeoBot Google-Extended GPTBot ImagesiftBot Magpie-crawler Omgili Omgilibot Peer39_crawler Peer39_crawler/1.0 PerplexityBot YouBot Googlebot Googlebot-Image Googlebot-News Googlebot-Video AdSense AdsBot-Google AdsBot Google-InspectionTool APIs-Google Feedfetcher-Google Google-NotebookLM facebookexternalhit Facebot meta-externalagent Facebook Meta *


Googlebot googlebot-image googlebot-mobile MSNBot Bingbot DuckDuckBot Twitterbot Yandex Slurp Teoma twiceler Gigabot Scrubby Robozilla Nutch ia_archiver baiduspider naverbot yeti yahoo-mmcrawler psbot asterias yahoo-blogs dotbot TweetmemeBot bitlybot *


Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot


* Googlebot-Image AhrefsBot AhrefsSiteAudit cortex BLEXBot Yandex


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Mediapartners-Google GPTBot Google-Extended FacebookBot CCBot anthropic-ai PerplexityBot Bytespider Applebot-Extended


Googlebot
Googlebot-Image
*
AdsBot-Google
MJ12bot
BLEXBot
Baiduspider
EasouSpider
Exabot
yandexbot
trovitbot
trovitBot
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
WebReaper
ZumBot
facebookexternalhit/1.1
facebookexternalhit/1.0
facebookexternalhit/*


Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)

Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://www.megaindex.com/?tab=linkAnalyze)

Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/)

Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://www.megaindex.com/)


* FacebookExternalHit Twitterbot Pinterestbot LinkedInBot TikTokSpider Storebot-Google Google-InspectionTool AmazonAdBot Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended Amazonbot meta-webindexer meta-externalagent meta-externalfetcher ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User cohere-training-data-crawler cohere-ai CCBot PanguBot PetalBot Bytespider Diffbot DuckAssistBot MistralAI-User Timpibot Webzio Webzio-Extended archive.org_bot ia_archiver ia_archiver-web.archive.org heritrix


*
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot-Extended
Bytespider
CCBot
Claude-Web
ClaudeBot
cohere-ai
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
Google-Extended
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
Meta-ExternalAgent
Meta-ExternalFetcher
omgili
omgilibot
PetalBot
Scrapy
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot


GPTBot Google-Extended ClaudeBot anthropic-ai PerplexityBot Perplexity-User CCBot


Googlebot
msnbot
Bingbot
SemrushBot
AhrefsBot
PetalBot
Sogou web spider
Sogou inst spider
contabo
Baiduspider


* Googlebot-Image Baiduspider ia_archiver GPTBot


* Googlebot Googlebot-image PerplexityBot OAI-SearchBot


* GPTBot ChatGPT-User PerplexityBot Perplexity-User SimpleFeed AmazonAdBot


Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)

Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)

Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)

Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)

Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)

Mozilla/4.0 (compatible; BullsEye; Windows 95)

Crescent Internet ToolPak HTTP OLE Control v.1.0


GPTBot SemrushBot AhrefsBot Bytespider ChatGPT CCBot ChatGPT-User anthropic-ai ClaudeBot Omgilibot Omgili FacebookBot Diffbot ImagesiftBot cohere-ai CLAUDE-WEB *


AmazonAdBot
Mediapartners-Google
EmailCollector
GagaRobot
vscooter
Roverbot*
mirago
psbot
MSIECrawler
WebCapture*
WebSauger*
Teleport*
Webwhacker*
Webzip*
WebCopier
Net Attache*
WebReaper*
SiteSnagger*
HTTrack*
*


Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/w4mwnpbXf3MFAbxOkJRw; +http://www.echobox.com)

Mozilla/5.0 (compatible; NewsRoom.BI/0.1; +http://www.newsroom.bi/bot.html)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36(KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 (compatible; mrfCompass-Booldog/1.0)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; mrfCompass-Marshall/1.0)

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Jukebox/1.0)


* GPTBot ChatGPT-User Google-Extended ClaudeBot anthropic-ai PerplexityBot Amazonbot Bytespider cohere-ai


* daumoa Sosospider rogerbot semetrical Googlebot-News AhrefsBot GPTBot OAI-SearchBot Applebot-Extended PerplexityBot Perplexity-ai CCBot YouBot ClaudeBot anthropic-ai Claude-Web grapeshot bingbot AmazonAdBot ozone Meta-ExternalAgent


*
AddSearchBot
AI2Bot
Ai2Bot-Dolma
aiHitBot
Amazonbot
Andibot
anthropic-ai
Applebot
Applebot-Extended
archive.org_bot
Awario
bedrockbot
bigsur.ai
Brightbot 1.0
Bytespider
CCBot
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
Google-Firebase
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
LinerBot
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
MistralAI-User
MistralAI-User/1.0
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
ShapBot
Sidetrade indexer bot
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YaK
YandexAdditional
YandexAdditionalBot
YouBot


*
BLEXBot
AhrefsBot
Vagabondo
SEOkicks-Robot
ia_archiver
archive.org_bot
special_archiver
MJ12bot
heritrix
netEstate NE Crawler
sistrix
WBSearchBot
QuerySeekerSpider
proximic
SiteExplorer
SemrushBot
SemrushBot-SA


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent ia_archiver


GPTBot ChatGPT-User Google-Extended ClaudeBot Anthropic-User-Agent PerplexityBot cohere-ai aicodebot YouBot *


*
Twitterbot
facebookexternalhit
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
Brightbot 1.0
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Crawlspace
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
imgproxy
ISSCyberRiskCrawler
Kangaroo Bot
Meta-ExternalAgent
Meta-ExternalFetcher
NewsNow
news-please
NovaAct
OAI-SearchBot
omgili
omgilibot
Operator
PanguBot
Perplexity-User
PerplexityBot
PetalBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent ia_archiver SemrushBot SemrushBot-SA SEOquake


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User ClaudeBot cohere-ai Diffbot FacebookBot GPTBot ImagesiftBot Meta-ExternalAgent Meta-ExternalFetcher Omgilibot PerplexityBot Timpibot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-Web PerplexityBot Google-Extended Applebot-Extended CCBot


* GPTBot Google-Extended Applebot-Extended ClaudeBot CCBot Bytespider


Cliqzbot
MJ12bot
rogerbot
dotbot
Alexibot
SurveyBot
Xenu's
Xenu's Link Sleuth 1.1c
AspiegelBot
omgili
*


GPTBot ChatGPT-User ClaudeBot Claude-Web PerplexityBot CCBot Amazonbot Bytespider *