Amazonbot
Applebot-Extended
Brightbot
ClaudeBot
Google-Extended
GPTBot
PerplexityBot
PetalBot
Scrapy
uptimerobot
viberbot
YaK
Yandex
Yeti
Applebot
Bingbot
Discordbot
DuckDuckBot
facebookexternalhit
Googlebot
Googlebot-Image
Googlebot-Video
Googlebot-News
Google-InspectionTool
LinkedInBot
msnbot
Pinterestbot
Screaming Frog SEO Spider
seznambot
Slurp
teoma
TelegramBot
Twitterbot
*


Googlebot Bingbot facebookexternalhit Google-Extended FacebookBot Discordbot *


*
Baiduspider
HaoSouSpider
Sogou web spider
Sogou inst spider
Sogou spider2


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* GPTBot ChatGPT-User Google-Extended Anthropic-AI Claude-Web CCBot PerplexityBot Cohere-ai


* GPTBot ClaudeBot anthropic-ai Google-Extended Applebot-Extended PerplexityBot Bytespider CCBot Copilot


LinkedInBot
Googlebot
Applebot
Bingbot
msnbot
Slurp
Googlebot-Image
baiduspider
Baiduspider
seznambot
Teoma
Yandex
Yeti
msnbot-media
Googlebot-News
Googlebot-Video
Googlebot-Mobile
Mediapartners-Google
AdsBot-Google
DuckDuckBot
daumoa
OrangeBot
OrangeBot-Collector
StackRambler
Twitterbot/1.0
Mail.RU_Bot
naverbot
vebidoobot
MJ12bot
360Spider
Sogou
rogerbot
deepcrawl
Neevabot
coccocbot-web
Zoombot
NewsWhip-LinkedIn-Bot
Cincobot
facebookexternalhit
outbrain
OAI-SearchBot
redditbot
Google-Extended
anthropic-ai
ChatGPT-User
ClaudeBot
Claude-Web
Claude-User
Claude-SearchBot
cohere-ai
GPTBot
Google-CloudVertexBot
PerplexityBot
Perplexity-User
DuckAssistBot
MyCentralAIScraperBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
quillbot.com
BLEXBot
CCBot
DataForSeoBot
news-please
omgili
omgilibot
magpie-crawler
peer39_crawler
Quora-Bot
Scrapy
SeekrBot
SeznamHomepageCrawler
Timpibot
Poseidon Research Crawler
Bytespider
*


Amazonbot Applebot-Extended Brightbot ClaudeBot Google-Extended GPTBot PerplexityBot PetalBot Scrapy uptimerobot viberbot YaK Yandex Yeti facebookexternalhit meta-externalads *


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


MJ12bot
Mediapartners-Google*
IsraBot
Orthogaffe
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
fast
wget
grub-client
k2spider
NPBot
WebReaper
SemrushBot
*


CCBot img2dataset Google-Extended anthropic-ai Claude-Web Omgilibot Omgili FacebookBot Bytespider magpie-crawler PerplexityBot Perplexity‑User *


*
Eluta.ca
MS Search 6.0 Robot
gsa-crawler


*
SeznamBot
msnbot
Bingbot
AdIdxBot
BingPreview
Yandex
GPTBot
Google-Extended
anthropic-ai
Claude-Web
Claudebot
CCBot
CC-Crawler
Common Crawl
CommonCrawl
Diffbot
FacebookBot
Bytespider
Omgilibot
Omgili
ImagesiftBot
Meltwater
Seekr
Grok


*
googlebot
Googlebot-Video
Applebot
bingbot
Baiduspider
Baiduspider-mobile
Baiduspider-video
Baiduspider-image
NaverBot
Y!J-WSC
YJ-WSC
Yeti
Yandex
YandexBot
YandexMobileBot
YandexVideo
YandexWebmaster
YandexSitelinks
SeznamBot
facebookexternalhit
GPTBot
ChatGPT-User
OAI-SearchBot
Google-Extended
ClaudeBot
Claude-User
Claude-SearchBot
PerplexityBot
AdsBot-Google
Twitterbot
Adidxbot
Yahoo Pipes 1.0
Facebot


2checkout
AASA-Bot
AccessibleWebBot
Adagiobot
adbeat_bot
ADmantX
AdometryBot
Adsense-Snapshot-Google
AdsTxtCrawler
Adyen
AhrefsBot
AhrefsSiteAudit
Akismet
Algolia Crawler
AmazonProductDiscovery
Aolbot-News
APIs-Google
AppInsights
Applebot
AppleNewsBot
Archive-It
archive.org_bot
ArchiveBot
ArchiveTeam-ArchiveBot
AspiegelBot
Audisto Crawler
Awario
Baidu-ADS
Baiduspider
Barkrowler
bingbot
BingPreview
bitlybot
bl.uk_ldfc_bot
BLP_bbot
bordermanager
botify
BrandVerity
Bravebot
BrightEdge Crawler
BrightEdge-Crawler
BUbiNG
CaliperBot
CapitalOneBot
Catchpoint
cf-qualys-scanner
Chrome-Lighthouse
CJBot
ClarityBot
Clickagy Intelligence Bot v2
Cloudflare-AI-Search
CloudflareBrowserRenderingCrawler
coccocbot
ContextualBot
CookieReports
Coveobot
CriteoBot
curly-
cXensebot
Datadog
DataForSeoBot
Daum
demandbase
Discordbot
DotBot
DoubleVerifyBot
Doximity-Pipeline
Dragonbot
Dratabot
DuckAssistBot
DuckDuckBot
DuckDuckGo-Favicons-Bot
DVBot
eContext
ecurly Webhook
EngageBDR
europarchive.org
evidon
ExaleadCloudview
FacebookBot
facebookcatalog
facebookexternalhit
Facebot
FastDAST
FeedBurner
Feedfetcher-Google
FlipboardProxy
FlippBot
Freshping
GeedoProductSearch
Google Favicon
Google News
Google Page Speed Insights
Google Wireless Transcoder
Google-Apps-Script
Google-Certificates-Bridge
Google-CWS
Google-Display-Ads-Bot
Google-Firebase
Google-HotelAdsVerifier


*
GPTBot
OAI-SearchBot
ChatGPT-User
PerplexityBot
Perplexity-User
ClaudeBot
Claude-SearchBot
Claude-User
Googlebot-Image
AdsBot-Google
Google-Extended
XoviBot
BLEXBot
DotBot
MJ12bot
Xenu's
Xenu's Link Sleuth 1.1c
Zeus
Zeus 32297 Webster Pro V2.9 Win32
Updownerbot
Blekkobot
008


Baiduspider 360Spider Sogouspider Yisouspider PetalBot GPTBot OAI-SearchBot anthropic-ai ClaudeBot PerplexityBot Google-Extended Applebot-Extended meta-externalagent DuckAssistBot ChatGPT-User Gemini-Deep-Research GoogleAgent-Mariner Google-NotebookLM CCBot Claude-User Claude-SearchBot Perplexity-User AI2Bot MistralAI-User Bytespider * Bingbot


* Googlebot LinkedInBot facebookexternalhit Twitterbot PinterestBot InstagramBot Slackbot WhatsApp redditbot Tumblr TikTokBot Snapchat


*
AdsBot-Google
Verity
ias_crawler
Mediapartners-Google
CriteoBot
yandex.com/bot
admantx-ussy04/3.2
TTD-Content
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
Bedrockbot
Bytespider
CCBot
ChatGLM
ChatGPT-User
ClaudeBot
Claude-User
Claude-SearchBot
Claude-Web
cohere-ai
cohere-training-data-crawler
Cotoyogi
DeepSeekBot
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
Gensparkbot
Google-CloudVertexBot
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
Grok
iaskspider/2.0
ICC-Crawler
ImagesiftBot
Img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
KunatoCrawler
magpie-crawler
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MistralAI
MistralAI-User
Momentumbot
OAI-SearchBot
omgili
Omgilibot
Other
PanguBot
Perplexity-ai
PerplexityBot
Perplexity-User
QualifiedBot
Scrapy
Seekr
Sidetrade indexer bot
TikTokSpider
Timpibot
Twitterbot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
WRTNBot
YouBot


gsa-crawler-www Atomz/1.0 * Googlebot AdsBot-Google AdsBot-Google-Mobile SearchmetricsBot


* ADmantX AlphaBot anthropic-ai AwarioRssBot AwarioSmartBot BLEXBot Buzzbot Bytespider CCBot ChatGPT-User claritybot Claude-Web ClaudeBot cohere-ai Diffbot FacebookBot FriendlyCrawler Google-Extended GPTBot huggingface ImagesiftBot img2dataset magpie-crawler Meltwater Neevabot news-please NewsNow Nutch omgili omgilibot panscient.com Perplexity-ai PerplexityBot PetalBot PiplBot scoop.it Scrapy Seekr SentiBot SeznamBot TurnitinBot YouBot ZumBot Claude-SearchBot OAI-SearchBot


Crescent Internet ToolPak HTTP OLE Control v.1.0

Mozilla/4.0 (compatible; BullsEye; Windows 95)


Baiduspider
Googlebot
MSNBot
Baiduspider-image
YoudaoBot
Sogou web spider
Sogou inst spider
Sogou spider2
Sogou blog
Sogou News Spider
Sogou Orion spider
ChinasoSpider
Sosospider
yisouspider
EasouSpider
*


* Googlebot AdsBot-Google Googlebot-Image OAI-SearchBot GPTBot ChatGPT-User PerplexityBot Google-Extended Yandex Sitereport proximic


* GPTBot CCBot SentiBot sentibot Google-Extended FacebookBot omgili omgilibot Amazonbot Bytespider


*
Meta-WebIndexer
Meta-ExternalAgent
Meta-ExternalFetcher
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot-Extended
Brightbot 1.0
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Crawlspace
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
Googlebot
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
Perplexity‑User
PetalBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot


*
AddSearchBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
Andibot
Anomura
atlassian-bot
Brightbot
BuddyBot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT-User
Claude-User
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-ai
cohere-training-data-crawler
Copilot
CopilotNative
CopilotSapphire
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
EtaoSpider
factset_spyderbot
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Firebase
Google-NotebookLM
GoogleAgent-Mariner
GoogleAgent-Shopping
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
iaskspider
IbouBot
ICC-Crawler
img2dataset
ImagesiftBot
imageSpider
imgproxy
ISSCyberRiskCrawler
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-externalagent
meta-externalfetcher
meta-webindexer
MistralAI-User
netEstate Imprint Crawler
omgili
Panscient
panscient.com
PanguBot
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
Scrapy
SemrushBot-SWA
ShapBot
Sidetrade indexer bot
TavilyBot
TerraCotta
Thinkbot
Timpibot
TwinAgent
VelenPublicWebCrawler
webzio-extended
wpbot
WRTNBot
xAI-Grok
YaK
YouBot
ZanistaBot


Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)

Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)


* OAI-SearchBot ChatGPT-User GPTBot ClaudeBot claude-web Claude-User PerplexityBot Perplexity-User Google-Extended Applebot Applebot-Extended DuckAssistBot CCBot Diffbot MistralAI-User YouBot DeepSeekBot


Adsbot-Google Alexabot AppleBot archive.org_bot Assaybot baiduspider bitlybot bingbot BingPreview ChatGPT-User Claude-SearchBot Claude-User coccocbot Discordbot DuckDuckBot facebookexternalhit Google-Site-Verification Google-Sitemaps Googlebot Googlebot-Image Googlebot-Mobile Googlebot-News Googlebot-Video gsa-crawler HatenaBlog-bot ia_archiver Mediapartners-Google msnbot NAVER OAI-SearchBot PerplexityBot PetalBot Pingdom Pinterest redditbot Slackbot Slurp snapchat Twitterbot wp.com Yandex YandexImages YandexVideoParser Yeti ZoomBot magpie-crawler *


*
Googlebot
AliyunSecBot
AmazonBot
anthropic-ai
Applebot-Extended
archive.org_bot
AudigentAdBot
AwarioRssBot
AwarioSmartBot
BLEXBot
Bytespider
CCBot
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
cohere-ai
DataForSeoBot
Diffbot
DuckAssistBot
EchoboxBot
FacebookBot
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
GPTBot
ImagesiftBot
Jetslide
magpie-crawler
Meta-ExternalAgent
meta-externalagent
Meta-ExternalFetcher
meta-externalfetcher
Meta-WebIndexer
meta-webindexer/1.1
MyCentralAIScraperBot
NewsNow
news-please
OAI-SearchBot
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Perplexity-User
Poseidon Research Crawler
quillbot.com
Quora-Bot
Scrapy
SeekrBot
SeznamHomepageCrawler
TaraGroup Intelligent Bot
Timpibot
TurnitinBot
ViennaTinyBot
YouBot
AmazonAdBot
facebookexternalhit
Twitterbot
RedditBot


Googlebot Slurp Yandex msnbot * CCBot sentibot Google-Extended FacebookBot omgili omgilibot Amazonbot ClaudeBot anthropic-ai ImagesiftBot Applebot-Extended TurnitinBot meta-externalagent YouBot


* Amazonbot Applebot-Extended Bytespider ClaudeBot FacebookBot GoogleOther GPTBot meta-externalagent


anthropic-ai ClaudeBot Claude-Web GPTBot ChatGPT-User OAI-SearchBot CCBot PerplexityBot Google-Extended Applebot-Extended Bytespider Amazonbot Meta-ExternalAgent cohere-ai *


AI2Bot
Ai2Bot-Dolma
AliyunSecBot
Amazonbot
amzn-searchbot
amzn-user
anthropic-ai
Applebot-Extended
Archive.org_bot
AwarioRssBot
AwarioSmartBot
Brightbot 1.0
Bytespider
CCBot
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
cohere-ai
cohere-training-data-crawler
Crawlspace
DataForSeoBot
Diffbot
DuckAssistBot
EchoboxBot
Exabot
FacebookBot
FriendlyCrawler
GPTBot
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
IAB-Tech-Lab
iaskspider/2.0
ICC-Crawler
img2dataset
ISSCyberRiskCrawler
ImagesiftBot
Kangaroo Bot
magpie-crawler
MistralAI-user
MyCentralAIScraperBot
NewsNow
news-please
OAI-SearchBot
omgili
omgilibot
PanguBot
Panscient
PiplBot
PerplexityBot
Perplexity-User
PetalBot
Poseidon Research Crawler
QuillBot
quillbot.com
Quora-Bot
SBIntuitionsBot
Scrapy
SeekrBot
SemrushBot-OCOB
SemrushBot-SWA
SeznamHomepageCrawler
Sidetrade indexer bot
TaraGroup Intelligence Bot
Timpibot
TurnitinBot
VelenPublicWebCrawler
ViennaTinyBot
Webzio-Extended
YandexAdditional
YandexAdditionalBot
YouBot
*
Googlebot-News


* googlebot-news Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot cohere-ai Diffbot FacebookBot FriendlyCrawler GPTBot ImagesiftBot meta-externalagent Meta-ExternalAgent OAI-SearchBot omgili omgilibot PerplexityBot AhrefsBot AhrefsSiteAudit Claude-User Claude-SearchBot AmazonAdBot


ChatGPT-User AmazonAdBot * CriteoBot/0.1 Amazonbot anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ClaudeBot Claude-Web cohere-ai DataForSeoBot Diffbot FacebookBot FriendlyCrawler Google-Extended GPTBot ImagesiftBot magpie-crawler Meta-ExternalAgent meta-externalagent NewsNow news-please omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot Quora-Bot Scrapy TurnitinBot


* Mediapartners-Google NewsNow CCBot TurnitinBot PetalBot MoodleBot FacebookBot Bytespider Mojeek JenkersBot Seekr YouBot Arquivo-web-crawler coccocbot-web SeznamBot PerplexityBot yacy anthropic-ai ClaudeBot Claude-SearchBot Claude-User AwarioRssBot AwarioSmartBot SentiOne ImageSift Applebot-Extended YandexAdditional YandexAdditionalBot scalepostAI Buck meta-externalagent Amazonbot amazon-QBusiness DuckAssistBot Google-CloudVertexBot Amzn-SearchBot AhrefsBot AhrefsSiteAudit


* Amazonbot magpie-crawler CCBot omgili omgilibot Claude-Web ClaudeBot anthropic-ai cohere-ai Bytespider PetalBot Scrapy Applebot-Extended GPTBot ChatGPT-User Google-Extended PerplexityBot Perplexity-User Google-CloudVertexBot meta-externalagent OAI-SearchBot YandexAdditional YandexAdditionalBot TurnitinBot Brightbot ApifyBot ApifyWebsiteContentCrawler Diffbot Diffbot-User ExaBot TavilyBot ShapBot YouBot FirecrawlAgent Amzn-SearchBot Amzn-User ProRataInc CloudflareBrowserRenderingCrawler AhrefsBot


* PerplexityBot OAI-SearchBot ChatGPT-User GPTBot Perplexity-User Claude-User ClaudeBot


* bingbot AdsBot-Google AdIdxBot Twitterbot Bytespider CCBot ChatGLM-Spider ClaudeBot PerplexityBot anthropic-ai AmazonBot meta-externalagent GPTBot Applebot-Extended OAI-SearchBot ChatGPT-User Claude-SearchBot Claude-User OAI-AdsBot


* GPTBot PerplexityBot ClaudeBot Applebot-Extended Bytespider CCBot Google-Extended Meta-ExternalAgent meta-externalagent


* Amazonbot magpie-crawler CCBot omgili omgilibot ClaudeBot Claude-Web anthropic-ai cohere-ai Bytespider PetalBot Scrapy Applebot-Extended GPTBot ChatGPT-User Google-CloudVertexBot Google-Extended PerplexityBot Perplexity-User meta-externalagent OAI-SearchBot YandexAdditional YandexAdditionalBot TurnitinBot Brightbot ApifyBot ApifyWebsiteContentCrawler Diffbot Diffbot-User ExaBot TavilyBot ShapBot YouBot FirecrawlAgent Amzn-SearchBot Amzn-User ProRataInc CloudflareBrowserRenderingCrawler AhrefsBot


* GPTBot OAI-SearchBot Google-Extended PerplexityBot ClaudeBot Claude-SearchBot anthropic-ai cohere-ai CCBot Diffbot Applebot-Extended Applebot YouBot MistralAI-User


* AdsBot-Google AdsIdxBot Google-InspectionTool facebookexternalhit LinkedInBot PerplexityBot Perplexity-User Canva-Slackapp-LinkExpanding Canva-Teamsapp-LinkExpanding GPTBot ClaudeBot CCBot Applebot-Extended anthropic-ai Claude-Web Bytespider meta-externalagent ImagesiftBot Timpibot Scrapy Ai2Bot Diffbot img2dataset ICC-Crawler FacebookBot VelenPublicWebCrawler FriendlyCrawler omgili Webzio-Extended Ai2Bot-Dolma Twitterbot


cisco-googlebot-enterprise
GPTBot
CCBot
Google-Extended
ClaudeBot
Googlebot
Googlebot Smartphone
Google AdsBot
Bingbot
*
Yandex


Applebot
Applebot-Extended
baiduspider
Bingbot
Discordbot
facebookexternalhit
Googlebot
Googlebot-Image
Google-InspectionTool
ia_archiver
LinkedInBot
msnbot
Naverbot
Pinterestbot
Screaming Frog SEO Spider
seznambot
Slurp
teoma
TelegramBot
Twitterbot
Yandex
Yeti
SnapchatAdsBot
SemrushBot
OAI-SearchBot
ChatGPT-User
ChatGPT-User/2.0
ChatGPT Agent
OpenAI
GPTBot
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
PerplexityBot
Perplexity-User
MistralAI-User
MistralAI-User/1.0
Meta-ExternalFetcher
Google-Extended
Google
Google-NotebookLM
NotebookLM
Gemini-Deep-Research
GoogleAgent-Mariner
Google-CloudVertexBot
Amazonbot
amazon-kendra
AmazonBuyForMe
NovaAct
CCBot
atlassian-bot
Cloudflare-AutoRAG
KlaviyoAIBot
ShapBot
Bravebot
DuckAssistBot
Devin
*


* GPTBot ChatGPT-User ClaudeBot Claude-Web anthropic-ai Google-Extended meta-externalagent PerplexityBot cohere-ai Bytespider


360Spider
404checker
404enemy
80legs
Abonti
Aboundex
Acunetix
ADmantX
AfD-Verbotsverfahren
AhrefsBot
AIBOT
AiHitBot
Aipbot
Alexibot
Alligator
AllSubmitter
AlphaBot
Anarchie
Apexoo
ASPSeek
Asterias
Attach
autoemailspider
BackDoorBot
Backlink-Ceck
backlink-check
BacklinkCrawler
BackStreet
BackWeb
Badass
Bandit
Barkrowler
BatchFTP
Battleztar Bazinga
BBBike
BDCbot
BDFetch
BetaBot
Bigfoot
Bitacle
Blackboard
Black Hole
BlackWidow
BLEXBot
Blow
BlowFish
Boardreader
Bolt
BotALot
Brandprotect
Brandwatch
Bubing
Buddy
BuiltBotTough
BuiltWith
Bullseye
BunnySlippers
BuzzSumo
Calculon
CATExplorador
CazoodleBot
CCBot
Cegbfeieh
CheeseBot
CherryPicker
ChinaClaw
Chlooe
Claritybot
Cliqzbot
Cloud mapping
coccocbot-web
Cogentbot
cognitiveseo
Collector
com.plumanalytics
Copier
CopyRightCheck
Copyscape
Cosmos
Craftbot
crawler4j
crawler.feedback
CrazyWebCrawler
Crescent
CSHttp
Curious
Custo
DatabaseDriverMysqli
DataCha0s
DBLBot
demandbase-bot
Demon
Deusu
Devil
Digincore
DigitalPebble
DIIbot
Dirbuster
Disco
Discobot


Googlebot Googlebot-Image Googlebot-Video Google-PageRenderer Google-InspectionTool Google-Safety Google-Site-Verification AdsBot-Google AdsBot-Google-Mobile Chrome-Lighthouse Storebot-Google Mediapartners-Google GoogleStructuredData Google-Read-Aloud bingbot Applebot DuckDuckBot Baiduspider Sogou MojeekBot facebookexternalhit Twitterbot Pinterestbot LinkedInBot WhatsApp TelegramBot Discordbot Snapchat Slackbot-LinkExpanding AmazonAdBot Amazonbot Amzn-SearchBot Amzn-User ClarityBot Citoid *


GPTBot ClaudeBot anthropic-ai Google-Extended Applebot-Extended ChatGPT-User OAI-SearchBot Claude-User Claude-SearchBot PerplexityBot Perplexity-User DeepSeekBot Amazonbot cohere-ai MistralAI-User OpenClaw CCBot Bytespider FacebookBot meta-externalagent ImagesiftBot Diffbot omgili omgilibot *


* AdsBot-Google-Mobile AdsBot-Google PetalBot dotbot AhrefsBot


360Spider
360Spider-Image
360Spider-Video
AdsBot-Google
AdsBot-Google-Mobile
AdsBot-Google-Mobile-Apps
adidxbot
Applebot
AppleNewsBot
Baiduspider
Baiduspider-image
Baiduspider-news
Baiduspider-video
bingbot
BingPreview
BublupBot
CCBot
Cliqzbot
coccoc
coccocbot-image
coccocbot-web
Daumoa
Dazoobot
DeuSu
DuckDuckBot
DuckDuckGo-Favicons-Bot
EuripBot
Exploratodo
Facebot
Feedly
Findxbot
Googlebot
Googlebot-Image
Googlebot-Mobile
Googlebot-News
Googlebot-Video
HaoSouSpider
ichiro
istellabot
JikeSpider
Lycos
Mail.Ru
Mediapartners-Google
MojeekBot
msnbot
msnbot-media
OrangeBot
Pinterest
Plukkie
Qwantify
Rambler
SeznamBot
Sosospider
Slurp
Sogou blog
Sogou inst spider
Sogou News Spider
Sogou Orion spider
Sogou spider2
Sogou web spider
SputnikBot
Teoma
Twitterbot
wotbox
yacybot
Yandex
YandexMobileBot
Yeti
YioopBot
yoozBot
YoudaoBot
*


*
Fast corporate crawler
008
magpie-crawler
magpie-crawler/1.1
MagpieRSS
psnsearch
HTTrack
Acrolinx-bot
amazon-Quick-on-behalf-of


*
meta-externalagent
AhrefsBot
deepcrawl
MS Search 6.0 Robot


* AdsBot-Google Googlebot-Image AdIdxBot MSNBot-Media GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User Google-Extended PerplexityBot Perplexity-User


Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot)

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)

Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot)

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 5.0 Robot)


* bingbot Googlebot Googlebot-image GPTBot Google-Extended


* Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot FacebookBot GPTBot Google-Extended GoogleOther PerplexityBot YouBot meta-externalagent ImagesiftBot cohere-ai Diffbot FriendlyCrawler img2dataset omgili omgilibot Claude-Web ChatGPT-User


gsa-crawler-www Atomz/1.0 * Googlebot AdsBot-Google AdsBot-Google-Mobile SearchmetricsBot


AddSearchBot
AwarioBot
AwarioRssBot
AwarioSmartBot
BLEXBot
BomboraBot
CheckLinks
Clickagy Intelligence Bot
cocolyzebot
DataForSeoBot
DomainStatsBot
DotBot
FindLinks
getlinkinfo.com
harsilbot
HubSpot Crawler
linkCheck
LinkWalker
MBCrawler
MJ12bot
Morningscore Bot
nooze-bot
OpenindexSpider
OpenWebSpider
PetalBot
Qwantify
Qwantify-prod
RankFlex
Rogerbot
Screaming Frog SEO Spider
SearchmetricsBot
SeekportBot
SeekrBot
SemrushBot-BA
SemrushBot-OCOB
SEO-Checker-bot
SeobilityBot
SeoSiteCheckup
SEOdiver Bot
SEOkicks-Robot
SEOstats
SiteAuditBot
trendictionbot
webgains-bot
www.deadlinkchecker.com
YaK
*


Baiduspider
360Spider
Googlebot
Sogou web spider
bingbot
smspider
HaosouSpider
YisouSpider
Bytespider
DeepSeek
OAI-SearchBot
ChatGPT-User
Kimi
Doubao
meta-externalagent
ChatGLM
*


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


* AI2Bot Bytespider FriendlyCrawler ICC-Crawler cohere-ai cohere-training-data-crawler Meta-ExternalAgent


AhrefsBot SemrushBot MegaIndex Googlebot Yandex *


GPTBot
ClaudeBot
Claude-User
Claude-SearchBot
CCBot
Google-Extended
Applebot-Extended
Facebookbot
Meta-ExternalAgent
Meta-ExternalFetcher
diffbot
PerplexityBot
Perplexity‑User
Omgili
Omgilibot
webzio-extended
ImagesiftBot
Bytespider
TikTokSpider
Amazonbot
Youbot
SemrushBot-OCOB
Petalbot
VelenPublicWebCrawler
TurnitinBot
Timpibot
OAI-SearchBot
ICC-Crawler
AI2Bot
AI2Bot-Dolma
DataForSeoBot
AwarioBot
AwarioSmartBot
AwarioRssBot
Google-CloudVertexBot
PanguBot
Kangaroo Bot
Sentibot
img2dataset
Meltwater
Seekr
peer39_crawler
cohere-ai
cohere-training-data-crawler
DuckAssistBot
Scrapy
Cotoyogi
aiHitBot
Factset_spyderbot
FirecrawlAgent
*


*
MJ12bot
DataForSeoBot
BLEXBot
AhrefsBot
barkrowler
Screaming Frog SEO Spider
ZoomBot
magpie-crawler
DotBot
rogerbot
SemrushBot
SiteAuditBot
SemrushBot-BA
SemrushBot-SI
SemrushBot-SWA
SplitSignalBot
SemrushBot-OCOB
JamesBOT
Oncrawl
AwarioRssBot
AwarioSmartBot
AwarioBot
serpstatbot
netEstate NE Crawler
panscient.com
AcademicBotRTU


*
googlebot
googlebot-image
GoogleOther
Googlebot-Video
Google-InspectionTool
AdsBot-Google
AdsBot-Google-Mobile
AdsBot-Google-Mobile-Apps
Storebot-Google
google-read-aloud
mediapartners-google
bingbot
msnbot
bingpreview
slurp
yahoo
baiduspider
Pinterestbot
Yeti
MojeekBot
360Spider
google-cloudvertexbot
duckduckbot
Applebot
flipboard
qwantbot
SeznamBot
proximic
admantx
thetradedesk
outbrain
ias_crawler
AmazonAdBot
pubmatic
smartologybot
parselybot
Screaming Frog SEO Spider
AhrefsBot
SemrushBot
SimilarWebBot
SISTRIX
botify
Chrome-Lighthouse
ChatGPT-User
GPTBot
OAI-SearchBot
facebookexternalhit
facebot
twitterbot
linkedinbot
snapchat
sentry
Iframely
Vocabtracker
EpvzCrawl6194680250
Citoid
ZoteroTranslationServer


Googlebot
Mediapartners-Google
Googlebot-Mobile
AdsBot-Google
AdsBot-Google-Mobile
bingbot
msnbot
ia_archiver
Alexabot
ScoutJet
Slurp
Twitterbot
Baiduspider
FacebookBot
facebookexternalhit
DuckDuckBot
ChatGPT-User
OAI-SearchBot
PerplexityBot
Claude-User
Claude-SearchBot
GPTBot
Google-Extended
ClaudeBot
sentibot
Yandex
AwarioRssBot
AwarioSmartBot
ImagesiftBot
CCBot
Applebot-Extended
anthropic-ai
Omgilibot
Omgili
Diffbot
Bytespider
Kangaroo Bot
cohere-ai
Meta-ExternalAgent
Meta-ExternalFetcher
Timpibot
Webzio-Extended
YouBot
*


* Twitterbot SemrushBot Amazonbot AhrefsBot anthropic-ai Applebot-Extended archive.org_bot ArchiveBot Arquivo-web-crawler AwarioRssBot AwarioSmartBot Bytespider CCBot Claude-Web ClaudeBot cohere-training-data-crawler DataForSeoBot Diffbot europarchive.org FacebookBot heritrix ia_archiver ia_archiver-web.archive.org ImagesiftBot LinkArchiver meta-externalagent meta-externalfetcher Nicecrawler omgili omgilibot PerplexityBot Primalbot web-archive-net.com.bot YandexBot


*
Baiduspider
Sogou web spider
Sogou inst spider
Sogou spider2
HaoSouSpider
Bytespider
AmazonAdBot
360Spider


*
Google-Extended
Mediapartners-Google
Spinn3r
Googlebot-News
Twitterbot
GPTBot
CCBot
Python-urllib
python-requests
Python-http-client
ChatGPT-User
Claude-Web
anthropic-ai
PerplexityBot
YouBot
ClaudeBot
Claude-User
Claude-SearchBot
OAI-SearchBot
DuckAssistBot
cohere-ai
Google-Apps
Google-Apps-Script
AppEngine-Google
Google-Cloud
Google-CloudVertexBot
AmazonBot
AmazonAdBot
ByteSpider
Meta-ExternalAgent
Meta-ExternalFetcher
Meta-ExternalAgent-Image
Applebot-Extended
Bytespider-Image
Feedly
FeedlyBot
FeedlyApp
MWFeedParser
comscore
Comscore
HubSpot
hubspot
Criteo
criteo-bot
Peer39_Crawler
rogerbot
linkdexbot
NTENT
AndersPinkBot
IndeedBot
MJ12bot
PetalBot
YisouSpider
360Spider
Qwantify
ToutiaoSpider
Screaming Frog SEO Spider
PRTG
FreshpingBot
Panopta
DatadogSynthetics
Rackspace
censys
burp
Burp
check_http
DotcomMonitor
WatchSumo
WormlyBot
CalibreBot
AudistoBot
Hatena
Hatena-Bookmark
EveryoneSocialBot
Diffbot
Netvibes
WebCEO
Postano
RebelMouse
Muck-Rack
InstapaperViewer
Twurly
LivelapBot
DatagnionBot
Linespider
Discourse
Medusa
pingback
WordPress
wp_ping
MauiBot
ltx71
WeSEE
halebot
BrightBot
UnityPlayer


* bingbot Yandex Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended Amazonbot meta-externalagent ClaudeBot PerplexityBot cohere-training-data-crawler OmgiliBot AI2Bot Bytespider TikTokspider DeepSeekBot


* ImagesiftBot Claude-SearchBot ClaudeBot Bytespider AwarioRssBot AwarioSmartBot dotbot AhrefsBot AhrefsSiteAudit MJ12bot SemrushBot-BA SemrushBot-ESI SemrushBot-FT SemrushBot-OCOB SemrushBot-SI SemrushBot-SWA SiteAuditBot BrightEdge proximic DataForSeoBot SeobilityBot


ai2bot
anthropic-ai
Amazonbot
Amazon-kendra
Applebot-extended
AutoGPT
BedrockBot
Bytespider
CCBot
Chatglm-spider
ChatGPT
ChatGPT-User
Claudebot
Claude-user
Claude-search
Claude-web
Cohere-ai
Cotoyogi
Deepseek
Deepseek Chat
Deepseekbot
DuckAssistBot
DuckDuckBot
FacebookBot
Geminiios
Google-Extended
Googleagent-manier
GPTBot
Grok
ia-archiver
iaskbot
iaskspider
ICC-crawler
Linerbot
Metaaib
Meta-externalagent
Meta-externalfetcher
Meta-llama
Mistralai-user
NovaAct
OAI-Searchbot
Pangubot
Perplexity
PerplexityBot
Perplexity-user
Phindbot
PoeBot
Qwen
Sogou
Timpibot
Tongyi
Wenxiaobai
Xinghuo
Youbot
Yuanbaobot
Googlebot
Googlebot Smartphone
Google AdsBot
Bingbot
*


* Googlebot Mediapartners-Google YandexBot YandexVideo msnbot-media Botify


* Googlebot Google-Extended ChatGPT-User Bingbot Slurp DuckDuckBot YahooJP PetalBot PerplexityBot DuckAssistBot Perplexity-User YouBot Claude-User Gemini-Deep-Research Claude-SearchBot OAI-SearchBot Google-Agent Google-CloudVertexBot Meta-WebIndexer Meta-ExternalFetcher xAI-Grok Grok-DeepSearch Mediapartners-Google AdsBot-Google AdsBot-Google-Mobile adidxbot Googlebot-Image msnbot-media OmniExplorer_Bot ia_archiver 008 JobdiggerSpider Cliqzbot GoogleOther omgili omgilibot Webzio Webzio-Extended Scrapy MyCentralAIScraperBot facebookexternalhit WhatsApp snapchat TelegramBot Twitterbot Slackbot-LinkExpanding GPTBot CCBot anthropic-ai FacebookBot AmazonBot Applebot-Extended Bytespider Baiduspider cohere-training-data-crawler FriendlyCrawler img2dataset Meta-ExternalAgent ClaudeBot DeepSeekBot GrokBot Diffbot AI2Bot PanguBot Timpibot


Googlebot Googlebot-Image Bingbot Google-Extended GPTBot OAI-SearchBot ChatGPT-User PerplexityBot ClaudeBot Claude-User Claude-SearchBot AdsBot-Google AdIdxBot Applebot Pinterestbot Twitterbot FacebookBot facebookexternalhit * Slurp Y!J-BRU/VSIDX Y!J-BRW Y!J-ASR Y!J-MMP/dscv Y!J-WSC Y!J-DLC YJ-BRU/VSIDX YJ-BRW YJ-ASR YJ-MMP/dscv YJ-WSC YJ-DLC Bytespider


* 008 AhrefsBot Applebot-Extended Bytespider CCBot ClaudeBot dotbot Google-Extended GPTBot linguatools MJ12bot Ocelli Riddlerbot ShopWiki sistrix sitebot WBSearchBot www.integromedb.org/Crawler


* AmazonAdBot anthropic-ai AwarioSmartBot Bytespider CCBot ClaudeBot DataForSeoBot Diffbot newspaper peer39_crawler Scrapy Timpibot


claritybot GPTBot Google-Extended CCBot ChatGPT-User anthropic-ai Omgilibot Omgili FacebookBot Bytespider *


OmtrBot/1.0 * SEOkicks coccocbot AhrefsBot SemrushBot trendictionbot serpstatbot startmebot HatenaBlog-bot Cincraw InfoTigerBot snapchat Superfeedr YaK digitalshadowsbot Magus Quora-Bot INETDEX-BOT DataForSeoBot SerendeputyBot Deskyobot MJ12bot BrightEdge ICC-Crawler


*
YisouSpider
Sogou web spider/4.0
Sogou web spider
sogou spider
AhrefsBot
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
Brightbot 1.0
Amazonbot
anthropic-ai
Claude-Web
ClaudeBot
Operator
GPTBot
cohere-training-data-crawler
cohere-ai
Meta-ExternalFetcher
Meta-ExternalAgent
FacebookBot
Google-Extended
GoogleOther
GoogleOther-Video
GoogleOther-Image
Google-CloudVertexBot
PerplexityBot
Perplexity-User
AI2Bot
Ai2Bot-Dolma
Applebot-Extended
Aranet-SearchBot
archive.org_bot
Arquivo-web-crawler
Bytespider
CCBot
Cincraw
Diffbot
DuckAssistBot
FriendlyCrawler
ia_archiver
ia_archiver-web.archive.org
ImagesiftBot
img2dataset
Kangaroo LLM
LinerBot
Magpie-crawler
omgili
omgilibot
PanguBot
Peer39_crawler
TerraCotta
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot


googlebot
GPTBot
PerplexityBot
Amazonbot
anthropic-ai
AwarioRssBot
AwarioSmartBot
Bytespider
CCBot
HuggingFaceBot
ChatGPT-User
ClaudeBot
Claude-Web
DataForSeoBot
Diffbot
FacebookBot
Google-Extended
Google-NotebookLM
Google-Agent
magpie-crawler
NewsNow
news-please
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
Scrapy
TurnitinBot
cohere-ai
Applebot-Extended
Meta-ExternalAgent
meta-externalagent
OAI-SearchBot
Claude-User
polymarket-news-bot
KalshiBot
polymarket-bot
OpenClaw
gold-ai-event-bridge
TradingBot
trading-pipeline
kalshi-bot
BivariateKalshiPipeline
Falcon Trading Platform
StockMarketAnalyser
FinanceApp
polyclaw-ingester
IsoxyTrader
Neo Trading Bot
MerlinChartWizard
investment-dashboard
*


Mozilla/4.0 (compatible; BullsEye; Windows 95)

Crescent Internet ToolPak HTTP OLE Control v.1.0


Googlebot AdsBot-Google AdsBot-Google-Mobile * YandexBot


* GPTBot OAI-SearchBot ChatGPT-User ClaudeBot anthropic-ai PerplexityBot Perplexity-User Google-Extended Applebot-Extended CCBot Meta-ExternalAgent cohere-ai Bytespider Amazonbot Googlebot Bingbot DuckDuckBot


BLEXBot Twitterbot * facebookexternalhit


AASA-Bot Amazonbot Googlebot Google-Extended Facebot Applebot Twitterbot LinkedInBot Feedly Pingdom Applebot-Extended Arquivo-web-crawler Bytespider Diffbot GPTBot ia_archiver ImagesiftBot JamesBOT OAI-SearchBot PerplexityBot PetalBot Scrapy ChatGPT-User Bravebot CriteoBot/0.1 ClaudeBot Claude-User Claude-SearchBot *


bingbot EtaoSpider GPTBot CCBot AmazonAdBot facebookexternalhit Mediapartners-Google *


Baiduspider
YisouSpider
Bytespider
msnbot
Googlebot
ToutiaoSpider
bingbot
DingTalkBot
Pinterestbot
sogou web spider
360Spider
Mediapartners-Google
*


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Adsbot-Google Mediapartners-Google Slurp


* Yandex AdsBot-Google facebookexternalhit Twitterbot


* CCBot Google-Extended GoogleOther anthropic-ai omgili omgilibot Bytespider cohere-ai magpie-crawler NewsNow PerplexityBot Perplexity-User Claude-Web ClaudeBot Diffbot FacebookBot Applebot-Extended meta-externalagent Meta-ExternalFetcher


* FacebookExternalHit Twitterbot Pinterestbot LinkedInBot TikTokSpider Storebot-Google Google-InspectionTool AmazonAdBot Google-Extended Google-CloudVertexBot GoogleOther Applebot-Extended Amazonbot meta-webindexer meta-externalagent meta-externalfetcher ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User cohere-training-data-crawler cohere-ai CCBot PanguBot PetalBot Bytespider Diffbot DuckAssistBot MistralAI-User Timpibot Webzio Webzio-Extended archive.org_bot ia_archiver ia_archiver-web.archive.org heritrix


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


*
Googlebot-News
Mediapartners-Google
CrystalSemanticsBot
archive-it.org
archive.org_bot
ArchiveBot
Arquivo-web-crawler
Authory
babel
bl.uk_ldfc_bot
bl.uk_ldfc_renderbot
bne.es_bot
bnf.fr_bot
ecoresearch
ecoresearchCrawler
Europarchive.org
kb.dk_bot
Kulturarw3
LivelapBot
mediacloud
mirrorweb
MSIECrawler
nationalarchives.gov.uk
netarkivet.dk
netEstate NE Crawler
Nicecrawler
nl-israel_iaharvester2026
nlnbot
nlncrawler
PageFreezer
SafeDNS
SmarshBot
special_archiver
tkl.iis.u-tokyo.ac.jp
TurnitinBot
webarchiv.cz
WordPress.com mShots
XY-Archive-Compliance-Archiver
XY-Archive-Compliance-Crawler
AhrefsBot
AI2Bot
Ai2Bot-Dolma
Amazonbot
amazon-kendra-
anthropic-ai
Applebot-Extended
bedrockbot
Bytespider
CloudflareBrowserRenderingCrawler
CCBot
ChatGLM-Spider
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
cohere-ai
Cotoyogi
DeepSeekBot
Diffbot
DuckAssistBot
EchoboxBot
FacebookBot
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleOther-Image
GoogleOther-Video
GPTBot
Grok
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
KunatoCrawler
LinerBot
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MistralAI-User
OAI-Operator
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
Perplexity-User
PetalBot
QualifiedBot
Scrapy
Seekr
Sidetrade indexer bot
TaraGroup Intelligent Bot
TikTokSpider
Timpibot
VelenPublicWebCrawler


* facebookexternalhit LinkedInBot Twitterbot GPTBot


anthropic-ai
AmazonBot
Claude
Claude-Web
claudeweb
Amzn-SearchBot
Amzn-User
AndiBot
cohere-ai
ClaudeBot
ChatGPT-User
Googlebot-News
GPTBot
CCBot
Google-Extended
DataForSeoBot
Diffbot
Diffbot-user
Kagibot
FacebookBot
magpie-crawler
NewsNow
news-please
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Scrapy
TurnitinBot
Applebot-Extended
Meta-ExternalAgent
meta-externalagent
OAI-SearchBot
Bytespider
Claude-User
Claude-SearchBot
AI2Bot
AhrefsBot
Ai2Bot-Dolma
Applebot
Arquivo-web-crawler
AspiegelBot
AwarioRssBot
AwarioSmartBot
Barkrowler
coccocbot-web
cohere-training-data-crawler
crawler4j
Crawlspace
dotbot
DuckAssistBot
FriendlyCrawler
Google-CloudVertexBot
https://hada.news
https://www.imediaethics.org
iaskspider/2.0
ICC-Crawler
icc-crawler
ICC Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
JenkersBot
Kangaroo Bot
LivelapBot
MauiBot
Meltwater
Meta-ExternalFetcher
MJ12bot
Mojeek
MojeekBot
MoodleBot
NeticleBot
Netvibes
PanguBot
PetalBot
SeekportBot
Seekr
SemrushBot
SeznamBot
Sidetrade Indexer Bot
Timpibot
Turnitin
VelenPublicWebCrawler
Webzio-Extended
yacy
yacybot
Yandex
YandexAdditional
YandexAdditionalBot
YandexBot
YandexImages
YandexNews
Baiduspider
Baiduspider-image
BaiduSpider-render
Baiduspider-video
Sogou Web Spider
yisouspider


*
googlebot
Googlebot-Video
Applebot
bingbot
Baiduspider
Baiduspider-mobile
Baiduspider-video
Baiduspider-image
NaverBot
Y!J-WSC
YJ-WSC
Yeti
Yandex
YandexBot
YandexMobileBot
YandexVideo
YandexWebmaster
YandexSitelinks
SeznamBot
facebookexternalhit
GPTBot
ChatGPT-User
OAI-SearchBot
Google-Extended
ClaudeBot
Claude-User
Claude-SearchBot
PerplexityBot
AdsBot-Google
Twitterbot
Adidxbot
Yahoo Pipes 1.0
Facebot


* ADmantX AlphaBot anthropic-ai AwarioRssBot AwarioSmartBot BLEXBot Buzzbot Bytespider CCBot ChatGPT-User claritybot Claude-Web ClaudeBot cohere-ai Diffbot FacebookBot FriendlyCrawler Google-Extended GPTBot huggingface ImagesiftBot img2dataset magpie-crawler Meltwater Neevabot news-please NewsNow Nutch omgili omgilibot panscient.com Perplexity-ai PerplexityBot PetalBot PiplBot scoop.it Scrapy Seekr SentiBot SeznamBot TurnitinBot YouBot ZumBot


Baiduspider 360Spider Yisouspider Sogouspider Googlebot Applebot Bingbot DuckDuckBot Naverbot Twitterbot Yandex Facebot facebookexternalhit facebookcatalog Google-Extended GPTBot ChatGPT-User ClaudeBot PerplexityBot Applebot-Extended MetaAI CCBot ugqaautomaticallybot


ia_archiver GPTBot OAI-SearchBot ChatGPT-User *


Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot Cohere-ai GPTBot Google-CloudVertexBot Img2dataset PanguBot PetalBot SentiBot YouBot magpie-crawler meta-externalagent * ChatGPT-User ChatGPT-User/2.0 Gemini-Deep-Research OAI-SearchBot Google-Extended applebot PerplexityBot bingbot


Adsbot-Google cludo OAI-SearchBot ChatGPT-User ChatGPT-User/2.0 ClaudeBot claude-web Claude-SearchBot Claude-User PerplexityBot Perplexity-User Google-Extended Googlebot Bingbot Amazonbot Applebot FacebookBot meta-externalagent LinkedInBot Bytespider DuckAssistBot cohere-ai CCBot *


Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com

Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com

CSS Validator: Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0


*
endeca
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot-Extended
bedrockbot
Bytespider
CCBot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
Cotoyogi
DeepSeekBot
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
Grok
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
KunatoCrawler
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MistralAI-User
OAI-Operator
OAI-SearchBot
omgili
omgilibot
PanguBot
Perplexity-User
PerplexityBot
PetalBot
QualifiedBot
Scrapy
Seekr
Sidetrade indexer bot
TikTokSpider
Timpibot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
WRTNBot


*
Google-Extended
AI2Bot
Amazonbot
Anthropic-ai
Applebot
Applebot-Extended
AwarioRssBot
AwarioSmartBot
Bytespider
CCBot
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cohere-ai
cohere-training-data-crawler
Cotoyogi
DataForSeoBot
Datenbank Crawler
DeepSeek
Devin
Diffbot
DuckAssistBot
FacebookBot
Factset_spyderbot
Gemini-Deep-Research
GoogleAgent-Mariner
ICC-Crawler
ImagesiftBot
Kangaroo Bot
Meta-ExternalAgent
Meta-ExternalFetcher
MistralAI-User
netEstate Imprint Crawler
NovaAct
omgili
Omgilibot
Operator
PanguBot
Peer39_crawler/1.0
Perplexity-User
PerplexityBot
PetalBot
Qwen
SemrushBot-OCOB
Timpibot
Twitterbot
Webzio-Extended
YouBot


AdsBot-Google AdsBot-Google-Mobile AdIdxBot Mediapartners-Google Ads-Naver Y!J-BRY Y!J-BRZ Googlebot Bingbot Yeti Y!J-ASR Y!J-WSC Slurp *


*
Facebot
facebookexternalhit
LinkedInBot
Twitterbot
GPTBot
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Crawlspace
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
Meta-ExternalAgent
Meta-ExternalFetcher
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
PetalBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot


Baiduspider
Baiduspider-video
Googlebot
YoudaoBot
Sogou web spider
Sogou inst spider
360Spider
bingbot
*


ClaudeBot Claude-User Claude-SearchBot OAI-SearchBot GPTBot ChatGPT-User PerplexityBot Perplexity-User Google-Agent Google-NotebookLM Gemini-Deep-Research DuckAssistBot YouBot meta-externalagent *


*
AhrefsBot
YandexBot
bingbot
Algolia Crawler
PetalBot


OAI-AdsBot OAI-SearchBot * Yandexbot LinkedInBot Twitterbot FacebookBot FacebookExternalHit Meta-ExternalAgent Meta-ExternalFetcher AdsBot-Google AdsBot-Google-Mobile AdIdxBot Bingbot GPTBot ClaudeBot Claude-Web CCBot Applebot-Extended PerplexityBot anthropic-ai cohere-ai Diffbot Bytespider


Yandex Googlebot * Mediapartners-Google YandexDirect


*
Mediapartners-Google
AdsBot-Google
AmazonAdBot
Bidswitchbot
CriteoBot
DoubleVerifyBot
DVBot
Grapeshot
Gumgum
IAS_admantx
IAS_crawler
IAS_wombles
Leikibot
Opebot-v
Peer39_crawler
Proximic
PubMatic Crawler Bot
SinceraSyntheticUser
SlickBot
TTD-Content


*
AddSearchBot
AI2Bot
Ai2Bot-DeepResearchEval
Ai2Bot-Dolma
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-extended
atlassian-bot
AutoRAG
AwarioSmartBot
AzureAI-SearchBot
Big Sur AI
bigsur.ai
Brandwatch
Bravebot
Brightbot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
Datenbank-Crawler
DeepSeekBot
Devin
Diffbot
Direqt Anomura
DuckAssistBot
Echobot Bot
ExaBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
GPTBot
iAsk
iAskBot
iaskspider
ICC Crawler
ICC-Crawler
ImageSiftBot
imageSpider
Kangaroo Bot
Kangaroo-Bot
KlaviyoAIBot
Kunato
laion-huggingface-processor
LCC
LINER Bot
LinerBot
LinkupBot
Manus-User
meta-externalagent
meta-externalfetcher
meta-webindexer
mistral.ai
MistralAI-User
netEstate Imprint Crawler
NovaAct
Novellum
Novellum AI Crawl
OAI-SearchBot
omgili
opencode
PanguBot
PeopleInc-DCipher-Scraper/1.1.0
Perplexity-User
PerplexityBot
PetalBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
SeekrBot
SemrushBot-OCOB
SemrushBotSwa
ShapBot
Spider
TavilyBot
TerraCotta
Timpibot
Trae
TwinAgent
Velen Crawler
VelenPublicWebCrawler
Webzio-Extended
Wrtn


* anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User Claude-Web ClaudeBot cohere-ai Diffbot FacebookBot Google-Extended GPTBot omgili


Amazonbot
Applebot-Extended
Brightbot
ClaudeBot
Google-Extended
GPTBot
PerplexityBot
PetalBot
Scrapy
uptimerobot
viberbot
YaK
Yandex
Yeti
Applebot
Bingbot
Discordbot
DuckDuckBot
facebookexternalhit
Googlebot
Googlebot-Image
Googlebot-Video
Googlebot-News
Google-InspectionTool
LinkedInBot
msnbot
Pinterestbot
Screaming Frog SEO Spider
seznambot
Slurp
teoma
TelegramBot
Twitterbot
*


*
Robozilla
Gigabot
Nutch
naverbot
yeti
Sogou web spider
YisouSpider
Applebot
Teoma
baiduspider
Googlebot
Googlebot-Image
Googlebot-Mobile
Googlebot-News
Googlebot-Video
Storebot-Google
Mediapartners-Google
AdsBot-Google
bingbot
msnbot
msnbot-media
adidxbot
Slurp
yahoo-mmcrawler
yandex
ia_archiver
Applebot-Extended
Google-Extended
GPTBot
ChatGPT-User
Omgilibot
Omgili
Diffbot
ImagesiftBot
cohere-ai
Amazonbot
Bytespider
CCBot
anthropic-ai
ClaudeBot
meta-externalagent


*
YandexBot
Googlebot
Bingbot
Baiduspider
OAI-SearchBot
Sogou
DuckDuckBot
MojeekBot
SeznamBot
Qwantify
Yeti
Slurp
360Spider
meta-externalagent
ChatGPT-User
ClaudeBot
PerplexityBot
GPTBot
GoogleOther
Timpibot
SBIntuitionsBot
Bytespider
AI2Bot
Amazon Q
Claude-User
CCBot
Diffbot
YandexGPT
MistralBot
cohere-ai
youBot
NeevaAI


AI2Bot Ai2Bot-Dolma aiHitBot Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler DuckAssistBot FacebookBot Google-Extended GoogleOther GoogleOther-Image GoogleOther-Video GPTBot img2dataset Meta-ExternalAgent MyCentralAIScraperBot omgili omgilibot Quora-Bot TikTokSpider YouBot AdsBot-Google AdsBot-Google-Mobile AdsBot-Google-Mobile-Apps *


* GPTBot CCBot Google-Extended PerplexityBot ClaudeBot anthropic-ai Applebot-Extended


*
Baiduspider
HaoSouSpider
Sogou web spider
Sogou inst spider
Sogou spider2


PetalBot
bedrock-agentcore-browser
Amazon-Bot
Amazonbot
AmazonProductDiscoveryBot
Amzn-SearchBot
Amzn-User
Arquivo-web-crawler
bnf.fr_bot
Bytespider
Claude-Extension
ClaudeBot
Claude-User
Claude-SearchBot
Comet
CloudFlare-AlwaysOnline
Cloudflare-Healthchecks
CloudflareBrowserRendering
Cloudflare-Radar
CCBot
DeepSeekBot
DuckAssistBot
archive.org_bot
ia_archiver
MistralAI-User
Moreover
Adbeat
Argus
Baiduspider
Baiduspider-image
Bitdefender
Blogtrottr
BloombergBot
BoardReader
checkmy.ws
Clickagy
CookieBot
cXensebot
DataObserver
DomainStatsBot
Embedly
Feedspotbot
FlipboardProxy
Getintent
Google-Apps-Script
GoogleDrive
Grapevine6
Gumgum
HaloBot
HubSpot Crawler
IbouBot
Inout
LinerBot
MJ12bot
Manus
Meltwater
Metajob
Muckrack
NetSeer
Netvibes
Onyphe
OpenClaw
Opoint
OutbrainBot
PlumAnalytics
ProSitemaps
PerplexityBot
Perplexity-User
proximic
PubMatic
Quantcastbot
SearchConcepts
Semantic-Visions
SentiBot
Sindup
Sirdata
SmartologyBot
Superfeedr
surlybot
Synaptive
TinEye
Trendiction
TurnitinBot
Ubermetrics
Vebidoo
Webcorp
Weborama
WebPageTest
YandexAccessibilityBot
YandexDirect
YandexDirectDyn
YaDirectFetcher
YandexFavicons
YandexMetrika
YandexMobileBot
YandexNews
YandexRCA
YandexRenderResourcesBot
YandexUserproxy
*


Yandex ChatGPT-User ChatGPT * Twitterbot


Googlebot Googlebot-Image Mediapartners-Google Googlebot-News Googlebot-Video AdsBot-Google AdsBot-Google-Mobile AdsBot-Google-Mobile-Apps Storebot-Google Bingbot AdIdxBot DuckDuckBot Slurp Qwantify Pinterest * buenibot EtaoSpider Yandex Baiduspider Baiduspider-image 360Spider MJ12bot CazoodleBot HaosouSpider ia_archiver Wget TurnitinBot HTTrack


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


*
Kangaroo Bot
Nutch
Omgili
Omgilibot
PanguBot
PetalBot
Timpibot
Webzio-Extended
AI2Bot
AI2Bot-Dolma
Bytespider
CCBot
Diffbot
FacebookBot
FriendlyCrawler
ICC-Crawler
GPTBot
Applebot-Extended
Claude-Web
Claudebot
cohere-ai
cohere-training-data-crawler
Google-Extended
Meta-ExternalAgent


Googlebot Googlebot-Image AdsBot-Google GoogleOther *


* Baiduspider OAI-SearchBot ChatGPT-User PerplexityBot Perplexity-User Claude-SearchBot Claude-User DuckAssistBot MistralAI-User YouBot PetalBot GPTBot ClaudeBot anthropic-ai Claude-Web CCBot Google-Extended Applebot-Extended Amazonbot meta-externalagent Meta-ExternalAgent FacebookBot Bytespider cohere-ai cohere-training-data-crawler PanguBot Diffbot FriendlyCrawler ImagesiftBot img2dataset omgili omgilibot Webzio-Extended Timpibot Brightbot DataForSeoBot DeepSeekBot iaskspider GrokBot


2checkout
AASA-Bot
AccessibleWebBot
Adagiobot
adbeat_bot
ADmantX
AdometryBot
Adsense-Snapshot-Google
AdsTxtCrawler
Adyen
AhrefsBot
AhrefsSiteAudit
Akismet
Algolia Crawler
AmazonProductDiscovery
Aolbot-News
APIs-Google
AppInsights
Applebot
AppleNewsBot
Archive-It
archive.org_bot
ArchiveBot
ArchiveTeam-ArchiveBot
AspiegelBot
Audisto Crawler
Awario
Baidu-ADS
Baiduspider
Barkrowler
bingbot
BingPreview
bitlybot
bl.uk_ldfc_bot
BLP_bbot
bordermanager
botify
BrandVerity
Bravebot
BrightEdge Crawler
BrightEdge-Crawler
BUbiNG
CaliperBot
CapitalOneBot
Catchpoint
cf-qualys-scanner
Chrome-Lighthouse
CJBot
ClarityBot
Clickagy Intelligence Bot v2
Cloudflare-AI-Search
CloudflareBrowserRenderingCrawler
coccocbot
ContextualBot
CookieReports
Coveobot
CriteoBot
curly-
cXensebot
Datadog
DataForSeoBot
Daum
demandbase
Discordbot
DotBot
DoubleVerifyBot
Doximity-Pipeline
Dragonbot
Dratabot
DuckAssistBot
DuckDuckBot
DuckDuckGo-Favicons-Bot
DVBot
eContext
ecurly Webhook
EngageBDR
europarchive.org
evidon
ExaleadCloudview
FacebookBot
facebookcatalog
facebookexternalhit
Facebot
FastDAST
FeedBurner
Feedfetcher-Google
FlipboardProxy
FlippBot
Freshping
GeedoProductSearch
Google Favicon
Google News
Google Page Speed Insights
Google Wireless Transcoder
Google-Apps-Script
Google-Certificates-Bridge
Google-CWS
Google-Display-Ads-Bot
Google-Firebase
Google-HotelAdsVerifier


Google-InspectionTool
Googlebot
Googlebot-Image
Googlebot-Mobile
Googlebot-Video
LinkedInBot
Twitterbot
facebookexternalhit
AdsBot-Google
BingPreview
Mediapartners-Google
STC-bot
Yahoo! Slurp
bingbot
OAI-SearchBot
ChatGPT-User
Claude-User
PerplexityUser
Perplexity-User
Claude-SearchBot
PerplexityBot
GPTBot
ClaudeBot
CCBot
Google-Extended
anthropic-ai
Claude-Web
Meta-ExternalAgent
Meta-ExternalFetcher
RuxitSynthetic
*


*
Googlebot-Image
Googlebot-News
Cision
Talkwater
Jetbot
kbcrawl
Newzbin
Qwam content intelligence
Youmag
Synthesio
trendybuzz
scoop.it
linkfluence
grub-client
ia_archiver-web.archive.org
k2spider
libwww
wget
5erue
adequat
adequat-systems
coexel
leadbox
mention
mytwip
opinion-tracker
proxem
score3
vecteurplus
verticalsearch
vsw
winello
Fetch
infoseek
MSIECrawler
Offline Explorer
sitecheck.internetseer.com
Teleport
TeleportPro
WebCopier
WebStripper
Zealbot
asknread.com
omgilibot
omgili
CCBot
Google-Extended
anthropic-ai
Claude-Web
ClaudeBot
Applebot-Extended
Webzio-Extended
Amazonbot
Timpibot
AI2Bot
cohere-training-data-crawler
DuckAssistBot
Kangaroo Bot
PanguBot
MistralAI-User
facebookbot
FacebookBot
5emeRue
ACQUIRE MEDIA
ACTIV Financial (CME Group)
AlphaSense
AmiSoftware
archive.org_bot
Archive-It
ArgClrInt
Ask n read
Augure
auramundi
AwarioRssBot
AwarioSmartBot
Barchart.com
BattleFin
Bernin IT
Blackboard Safeassign
BLP_bbot
bluematrix
Brandwatch
Briefcase.news
Buck
Bytespider
CCBo
CikisiBot
Coexel
cohere-ai
Comtex News Network
ConveraCrawler
Copyright Licensing Agency
Corporama
D&B Hoovers
Data Expression
Data Observer
Dataminr
Dealogic
Diffbot


*
facebookexternalhit
LinkedInBot
Twitterbot
YouBot
YandexAdditional
YandexAdditionalBot
Applebot-Extended
Amazonbot
amazon-kendra
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
meta-externalagent
Meta-ExternalAgent
ActiveComplyBot
CriteoBot
GumGumBot
TaraGroup Intelligent Bot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


*
EtaoSpider
Twitterbot
AmazonAdBot
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok


* Bingbot Yandex Googlebot YandexAdditional


* NextGenSearchBot GeedoShopProductFinder ImagesiftBot EdisterBot Ezooms SWEBot discobot SemrushBot grapeshot BUbiNG Bytespider CCBot ClaudeBot Diffbot FacebookBot meta-externalagent Google-Extended GPTBot omgili Amazonbot


* Swiftbot phantomjs phantomJS DatadogSynthetics


* Twitterbot Amazonbot AmazonBot anthropic-ai Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web Claude-User Claude-SearchBot cohere-ai DataForSeoBot Diffbot FacebookBot Google-Extended GPTBot magpie-crawler Meta-ExternalAgent meta-externalagent MistralAI-User NagetBot NewsNow news-please OAI-SearchBot omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot Perplexity-User Scrapy TurnitinBot


Anthropic-ai ClaudeBot Bingbot BingAI BingPreview ChatGPT-User CopilotBot Googlebot Google-Extended GPTBot LlamaBot OAI-SearchBot PerplexityBot xAI-Bot Amazonbot FirecrawlAgent AndiBot ExaBot PhindBot YouBot PetalBot DeepSeekBot


* GPTBot CCBot PerplexityBot Perplexity-User OAI-SearchBot ChatGPT-User


Amazonbot PerplexityBot Perplexity-User ClaudeBot Claude-User GPTBot ChatGPT-User FacebookExternalHit FacebookBot Twitterbot LinkedInBot Bingbot *


* Googlebot LinkedInBot facebookexternalhit Twitterbot PinterestBot InstagramBot Slackbot WhatsApp redditbot Tumblr TikTokBot Snapchat


* 008 FeedBurner/1.0 GPTBot CCBot ChatGPT-User Google-Extended


GPTBot ChatGPT-User OAI-SearchBot anthropic-ai ClaudeBot GoogleOther Google-Extended Googlebot Googlebot-Image Applebot Bingbot PerplexityBot ScreamingFrogCrawler Brightbot *


* CCBot GPTBot anthropic-ai ClaudeBot Claude-Web cohere-ai PerplexityBot Amazonbot Applebot-Extended Bytedance Timpibot Claude-User Claude-SearchBot


Amazonbot
Applebot
GPTBot
A6-Indexer
AhrefsBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
Anchor Browser
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot-Extended
Baiduspider
betaBot
BLEXBot
botify
brands-bot-logo
Brightbot
Bytespider
calculon
CCBot
ChatGLM-Spider
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
collection@infegy.com
Cotoyogi
Crawl4AI
CrunchBot
deepcrawl
DeepSeekBot
Diffbot
DirBuster
DTS
DuckAssistBot
EasouSpider
FacebookBot
FirecrawlAgent
FriendlyCrawler
GlutenCrawler
Google-CloudVertexBot
Google-Extended
GrapeshotCrawler
iaskspider
ICC-Crawler
ImagesiftBot
img2dataset
JackCrawl
Jetslide
linkapedia_bot
Mediatoolkitbot
MegaIndex
meta-externalagent
meta-externalfetcher
Microsoft BITS
Microsoft URL Control
Microsoft-WebDAV-MiniRedir
MistralAI-User
news-please
Novellum
NTENTbot
Nutch
omgili
Omgilibot
OpenLinkProfiler
Perplexity-User
PerplexityBot
PetalBot
Primalbot
python-requests
Re-Animator
Riddler
SafeDNSBot
SBIntuitionsBot
Scrapy
SearchAtlas
SemanticScholarBot
SemrushBot
SentiBot
Sogou
Spider-Pig
sqlmap
StorygizeBot
Test Certificate Info
TikTokSpider
Timpibot
Turnitin
uipbot
vBSEO
VegeBot
VelenPublicWebCrawler
Web Link Validator
WebCruiser


* bingbot GPTBot ChatGPT-User OAI-Searchbot PerplexityBot Perplexity-User


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


*
AddSearchBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
Andibot
Anomura
atlassian-bot
Brightbot
BuddyBot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT-User
Claude-User
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-ai
cohere-training-data-crawler
Copilot
CopilotNative
CopilotSapphire
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
EtaoSpider
factset_spyderbot
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Firebase
Google-NotebookLM
GoogleAgent-Mariner
GoogleAgent-Shopping
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
iaskspider
IbouBot
ICC-Crawler
img2dataset
ImagesiftBot
imageSpider
imgproxy
ISSCyberRiskCrawler
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-externalagent
meta-externalfetcher
meta-webindexer
MistralAI-User
netEstate Imprint Crawler
omgili
Panscient
panscient.com
PanguBot
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
Scrapy
SemrushBot-SWA
ShapBot
Sidetrade indexer bot
TavilyBot
TerraCotta
Thinkbot
Timpibot
TwinAgent
VelenPublicWebCrawler
webzio-extended
wpbot
WRTNBot
xAI-Grok
YaK
YouBot
ZanistaBot


* OAI-SearchBot ChatGPT-User Claude-SearchBot PerplexityBot GPTBot ClaudeBot Google-Extended Applebot-Extended CCBot


*
Googlebot
Googlebot-Video
Googlebot-Image
Googlebot-News
Storebot-Google
Google-InspectionTool
Applebot
Bingbot
BingPreview
Slurp
DuckDuckBot
Baiduspider
Baiduspider-mobile
Baiduspider-video
Baiduspider-image
NaverBot
Y!J-WSC
YJ-WSC
Yeti
Yandex
YandexBot
YandexMobileBot
YandexVideo
YandexWebmaster
YandexSitelinks
SeznamBot
PetalBot
Sogou
Qwantify
Ecosia
360Spider
Coc Coc
Qwantbot
Botify
SemrushBot
Screaming Frog SEO Spider
DataForSeoBot
ia_archiver
Facebot
Twitterbot
facebookexternalhit
instagram
Threads
TikTok
Pinterest
Pinterestbot
redditbot
WhatsApp
Snapchat
TelegramBot
Viber
KakaoTalk
Line
AdsBot-Google
AdsBot-Google-Mobile
AdsBot-Google-Mobile-Apps
Mediapartners-Google
Google-Ads-Creatives-Assistant
FacebookAdPreview
meta-externalads
Adidxbot
Bytespider
IAS-Crawler
DoubleVerifyBot
Moatbot
TheTradeDesk
TTDCrawler
CriteoBot
GPTBot
ChatGPT-User
OAI-SearchBot
Google-Extended
ClaudeBot
Claude-User
Claude-SearchBot
PerplexityBot
Applebot-Extended
Meta-ExternalAgent
Meta-ExternalFetcher
Amazonbot
CCBot
Google-Agent
GrokBot
YouBot
DuckAssistBot
DeepSeekBot


Googlebot Bingbot YandexBot Baiduspider OAI-Searchbot ChatGPT-User GPTBot meta-externalagent PerplexityBot anthropic-ai ClaudeBot cohere-ai ImagesiftBot *


AhrefsBot
AI2Bot
Ai2Bot-Dolma
aiHitBot
Andibot
Applebot-Extended
Awario
Baiduspider
bedrockbot
BLEXBot
Brightbot 1.0
Bytespider
CCBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawlspace
DataForSeoBot
Datenbank Crawler
Devin
Diffbot
dotbot
DuckAssistBot
Echobot Bot
EchoboxBot
Factset_spyderbot
FirecrawlAgent
Flamingo_SearchEngine
FriendlyCrawler
ia_archiver
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
LCC
magpie-crawler
MistralAI-User
MistralAI-User/1.0
MJ12bot
MyCentralAIScraperBot
netEstate Imprint Crawler
NovaAct
omgili
omgilibot
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poseidon Research Crawler
QualifiedBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot
SemrushBot-OCOB
SemrushBot-SA
SemrushBot-SWA
sentibot
Sidetrade indexer bot
SirdataBot
sogou spider
SummalyBot
TerraCotta
Thinkbot
TikTokSpider
Timpibot
TurnitinBot
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
YandexAdditional
YandexAdditionalBot
Yeti
YouBot
Googlebot-News
*


*
Teleport
WebZIP
CherryPicker
ERNIE-Bot
BlowFish/1.0
grub
Bytespider
SiteSnagger
Titan
Barkrowler
008
Alexibot
BackDoorBot/1.0
Bullseye/1.0
ExtractorPro
Kenjin Spider
WWW-Collector-E
Sogou
Yandex
Mail.RU_Bot
MegaIndex
Szukacz/1.4
BotALot
BunnySlippers
BuiltBotTough
CheeseBot
Crescent
EroCrawler
Foobot
Harvest/1.5
InfoNaviRobot
JennyBot
Mister PiX
moget
NetAnts
NICErsPRO
ProPowerBot/2.14
Radiation Retriever
RepoMonkey
RepoMonkey Bait & Tackle/v1.01
True_Robot
URL_Spider_Pro
Web Image Collector
WebAuto
WebBandit
WebCopier
Website Quester
Webster Pro
WebStripper
Wget
Zeus
dotbot
Rogerbot
ia_archiver
MJ12bot
Xenu
AhrefsBot
CCBot
SemrushBot
DataForSeoBot
XoviBot
Screaming Frog SEO Spider
Keyword Density/0.9
LinkextractorPro
LinkScan/8.1a Unix
BLEXbot
spbot


* Googlebot-Image zs-crawler zoho-scrapy


Googlebot APIs-Google Mediapartners-Google msnbot bingbot BingPreview MicrosoftPreview Slurp DuckDuckbot baiduspider seznambot naverbot teoma Yandex ScoutJet GurujiBot Exabot Solbot FAST VoilaBot daumoa ArchitextSpider Twitterbot SocSciBot ia_archiver coccoc Browsershots SiteArc Cliqzbot proximic OnCrawlBot Botify Googlebot-Image Googlebot-Video Googlebot-News Google-InspectionTool Gemini-Deep-Research OAI-SearchBot ChatGPT-User PerplexityBot Perplexity‑User Claude-SearchBot Claude-User DuckAssistBot facebookexternalhit LinkedInBot Bluesky Discordbot TelegramBot Slackbot Applebot YouBot MistralAI-User Kagibot TavilyBot Applebot-Extended GPTBot Google-Extended *


* AdsBot-Google AdsIdxBot Google-InspectionTool facebookexternalhit Claude-SearchBot Claude-User Perplexity-User Applebot-Extended GPTBot Google-Extended Bytespider ClaudeBot CCBot PerplexityBot Twitterbot


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


Fasterfox Nutch spock OmniExplorer_Bot MJ12bot TurnitinBot BecomeBot genieBot MLBot GPTBot cohere-ai anthropic-ai CCBot FacebookBot Google-Extended omgilibot omgili GleanSearchCrawlerBot amazon-kendra * CHATGPT-User OAI-SearchBot PerplexityBot Claude-SearchBot


* bingbot AdsBot-Google AdIdxBot Twitterbot Bytespider CCBot ChatGLM-Spider ClaudeBot PerplexityBot anthropic-ai AmazonBot meta-externalagent GPTBot Applebot-Extended


* Twitterbot Amazonbot Anthropic-ai AwarioRssBot AwarioSmartBot Bytespider CCBot Cohere-ai DataForSeoBot ImagesiftBot Magpie-crawler Omgili Omgilibot Peer39_crawler Peer39_crawler/1.0 YouBot


*
Googlebot-News
Mediapartners-Google
CrystalSemanticsBot
archive-it.org
archive.org_bot
ArchiveBot
Arquivo-web-crawler
Authory
babel
bl.uk_ldfc_bot
bl.uk_ldfc_renderbot
bne.es_bot
bnf.fr_bot
ecoresearch
ecoresearchCrawler
Europarchive.org
kb.dk_bot
Kulturarw3
LivelapBot
mediacloud
mirrorweb
MSIECrawler
nationalarchives.gov.uk
netarkivet.dk
netEstate NE Crawler
Nicecrawler
nl-israel_iaharvester2026
nlnbot
nlncrawler
PageFreezer
SafeDNS
SmarshBot
special_archiver
tkl.iis.u-tokyo.ac.jp
TurnitinBot
webarchiv.cz
WordPress.com mShots
XY-Archive-Compliance-Archiver
XY-Archive-Compliance-Crawler
AhrefsBot
AI2Bot
Ai2Bot-Dolma
Amazonbot
amazon-kendra-
anthropic-ai
Applebot-Extended
bedrockbot
Bytespider
CloudflareBrowserRenderingCrawler
CCBot
ChatGLM-Spider
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
cohere-ai
Cotoyogi
DeepSeekBot
Diffbot
DuckAssistBot
EchoboxBot
FacebookBot
FriendlyCrawler
Gemini-Deep-Research
Google-CloudVertexBot
Google-Extended
GoogleOther-Image
GoogleOther-Video
GPTBot
Grok
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kangaroo Bot
KunatoCrawler
LinerBot
Meltwater
Meta-ExternalAgent
Meta-ExternalFetcher
MistralAI-User
OAI-Operator
OAI-SearchBot
omgili
omgilibot
PanguBot
PerplexityBot
Perplexity-User
PetalBot
QualifiedBot
Scrapy
Seekr
Sidetrade indexer bot
TaraGroup Intelligent Bot
TikTokSpider
Timpibot
VelenPublicWebCrawler


* gsa-crawler OmniExplorer_Bot Googlebot-Mobile msnbot-mobile YahooSeeker/M1A1-R2D2


Mozilla/4.0 (compatible; BullsEye; Windows 95)

Crescent Internet ToolPak HTTP OLE Control v.1.0


GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Claude-User
Claude-SearchBot
Claude-Web
anthropic-ai
PerplexityBot
Perplexity-User
Google-Extended
Gemini-Deep-Research
Google-CloudVertexBot
Applebot-Extended
DuckAssistBot
MistralAI-User
cohere-ai
cohere-training-data-crawler
CCBot
FacebookBot
meta-externalagent
Meta-ExternalFetcher
Amazonbot
YouBot
Webzio
Webzio-Extended
omgili
Omgilibot
Diffbot
Bytespider
AI2Bot
Kangaroo Bot
PanguBot
PetalBot
SemrushBot-OCOB
*


Twitterbot facebookexternalhit LinkedInBot *


* Googlebot-News Twitterbot Mediapartners-Google AmazonAdBot Bytespider ClaudeBot Diffbot discobot GPTBot ImagesiftBot omgili omgilibot


Googlebot * facebookexternalhit slackbot-linkexpanding slack-imgproxy twitterbot whatsapp skypeuripreview bf-directline LinkedInBot snippet wakeletlinkexpander


* bingbot AdsBot-Google AdIdxBot Twitterbot Bytespider CCBot ChatGLM-Spider ClaudeBot PerplexityBot anthropic-ai AmazonBot meta-externalagent GPTBot Applebot-Extended OAI-SearchBot ChatGPT-User Claude-SearchBot Claude-User


* Mediapartners-Google Adsbot-Google Twitterbot


Amazonbot PerplexityBot Perplexity-User ClaudeBot Claude-User GPTBot ChatGPT-User FacebookExternalHit FacebookBot Twitterbot LinkedInBot Bingbot *


*
Googlebot
Googlebot-Image
Googlebot-Mobile
Googlebot-Video
Googlebot-News
Bingbot
msnbot
bingbot-mobile
Baiduspider
Baiduspider-image
Baiduspider-video
YandexBot
YandexBot-Image
YandexBot-Mobile
Sogou web spider
Sogou Pic Spider
360Spider
HaosouSpider
Sosoimagespider
Sosospider


* AmazonAdBot Amazonbot anthropic-ai Applebot Applebot-Extended ArcSpanScraper AwarioRssBot AwarioSmartBot BingBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User Claude-Web cohere-ai DataForSeoBot Diffbot DuckAssistBot FacebookBot Google-Extended Googlebot GPTBot magpie-crawler Meta-ExternalAgent Meta-ExternalFetcher MistralAI-User NewsNow news-please omgili omgilibot OAI-SearchBot Operator PetalBot PerplexityBot Perplexity-User ProRataInc Quora-Bot Scrapy TerraCotta TimpiBot TurnitinBot archive.org_bot


* AdsBot-Google CCBot ChatGPT-User ClaudeBot Doximity GPTBot


*
Googlebot-News
GPTBot
Google-Extended
CCBot
anthropic-ai
Omgilibot
FacebookBot
cohere-ai
PetalBot
Yandex
PerplexityBot
Amazonbot
AhrefsBot
archive.org_bot
ArchiveBot
Arquivo-web-crawler
AwarioRssBot
AwarioSmartBot
BacklinkCrawler
BIXOCRAWLER
BLEXBot
Bytespider
Cliqzbot
ConveraCrawler
crawler4j
CrystalSemanticsBot
DataForSeoBot
Diffbot
discobot
discoverybot
dloader(NaverRobot)
dotbot
Download Ninja
DuckAssistBot
europarchive.org
Exabot
ExtractorPro
Fetch
Flamingo_SearchEngine
FriendlyCrawler
Gigabot
GumGum Bot
ia_archiver
ia_archiver-web.archive.org
ImagesiftBot
IstellaBot
Jetbot
JikeSpider
Jyxobot
k2spider
Kangaroo Bot
kbcrawl
LexxeBot/1.0
LinkArchiver
LinkextractorPro
Livelapbot
magpie-crawler
meta-externalfetcher
MJ12bot
Moreoverbot
MSIECrawler
nabot
NaverBot
NerdByNature.Bot
netEstate NE Crawler
nNetSeer crawler
NextGenSearchBot
Nicecrawler
NPBot
nutch
Openbot
PanguBot
peer39_crawler
peer39_crawler/1.0
PiplBot
Pixray-Seeker
Primalbot
psbot
QuerySeekerSpider
Quora-Bot
rogerbot
Scrapy
searchpreview
SemrushBot
SEOENGBot
SEOkicks-Robot
SiteBot
SiteBot/0.1
Slurp
Sosospider
spbot
Timpibot
trendictionbot
trovitBot
True_Robot
TurnitinBot
UbiCrawler
uMBot-LN
URL_Spider_Pro


*
bingbot
AddSearchBot
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
AliyunSecBot/Aliyun
amazon-kendra
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Amzn-User
Andibot
Anomura
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-Extended
Aranet-SearchBot
atlassian-bot
Autn-WKOOP
Awario
AzureAI-SearchBot
barkrowler
bedrockbot
bigsur.ai
Bravebot
Brightbot
Brightbot 1.0
BuddyBot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT Agent
ChatGPT-User
Claude-Code
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
CloudVertexBot
Code
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Crawlspace
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
dobot
DuckAssistBot
Echobot Bot
EchoboxBot
ExaBot
FacebookBot
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
Gemini-Deep-Research
Google-Agent
Google-CloudVertexBot
Google-Extended
Google-Firebase
Google-Gemini-CLI
Google-NotebookLM
GoogleAgent-Mariner
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
HawaiiBot
HenkBot
iAskBot
iaskspider
iaskspider/2.0
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
img2dataset
ISSCyberRiskCrawler
kagi-fetcher
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
LAIONDownloader
LCC
LinerBot
Linguee Bot
LinkupBot
Manus-User
meta-externalagent
meta-externalfetcher


GPTBot CCBot anthropic-ai AmazonAdBot Google-CloudVertexBot AppleBot Google-Extended Applebot-Extended ClaudeBot Amazonbot Meta-ExternalAgent PetalBot Yandex Bytespider Scrapy Diffbot-User YouBot Timpibot *


* Googlebot bingbot Yeti Twitterbot Livelapbot Megalodon ia_archiver


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Pinterest Pinterestbot ChatGPT-User OAI-SearchBot GPTBot CCBot news-please anthropic-ai ClaudeBot Claude-Web cohere-ai ImagesiftBot FriendlyCrawler Quora-Bot omgilibot omgili PerplexityBot Perplexity-User Google-Extended


* Adsbot-Google Mediapartners-Google zoomRank/2.0 Google-Display-Ads-Bot OAI-SearchBot GPTBot ChatGPT-User PerplexityBot AmazonAdBot


* Baiduspider GPTBot CCBot PerplexityBot Bytespider


* Googlebot AdsBot-Google google-hoteladsverifier TurnitinBot NPBot-1/2.0 NPBot psbot Mediapartners-Google


Baiduspider Bingbot DuckDuckBot Slurp Googlebot Yandex *


* GPTBot Google-Extended anthropic-ai ClaudeBot FacebookBot Applebot-Extended meta-externalagent cohere-ai CCBot omgili Diffbot omgilibot


*
008
AddThis.com
admantx
AhrefsBot
BDCbot
Bender
BIXOCRAWLER
bl.uk_lddc_bot
BLEXBot
BUbiNG
Cliqzbot
CNCDialer
crawler4j
CrystalSemanticsBot
CyberAlert
DigExt
discobot
discoverybot
dloader
dloader(NaverRobot)
DOC
dotbot
Download Ninja
DTS Agent
Exabot
Ezooms
FairShare
Fetch
Flamingo_SearchEngine
Genieo
Gigabot
grub-client
Heritrix
heritrix/3.3.0
HTTrack
ia_archiver
integromedb
IstellaBot
JikeSpider
Jyxobot
k2spider
Kimengi
Kimengi/nineconnections.com
larbin
LexxeBot/1.0
libwww
linko
Livelapbot
magpie-crawler
Maxthon
MetaURI
Microsoft.URL.Control
MJ12bot
Moreover
Moreoverbot
MSIECrawler
nabot
NaverBot
NerdByNature.Bot
netEstate NE Crawler
NetSeer crawler
Newscan
NextGenSearchBot
NPBot
nutch
Offline Explorer
omgilibot
Orthogaffe
PiplBot
Pixray-Seeker
proximic
psbot
QuerySeekerSpider
rogerbot
seokicks
SEOkicks-Robot
SiteBot
SiteBot/0.1
sitecheck.internetseer.com
SiteSnagger
Slurp
sogou
Sosospider
spbot
Spinn3r
Teleport
TeleportPro
trendictionbot
trovitBot
TurnitinBot
UbiCrawler
uMBot-LN
UnisterBot
UniversalFeedParser
WBSearchBot
WebCopier
WebReaper
WebStripper
WebZIP


Baiduspider 360Spider Sogouspider Yisouspider PetalBot Bytespider


*
gsa-garmin
Better Uptime Bot
DataForSeoBot
ecoresearchCrawler
EyeMonIT Uptime Bot
omgili
omgilibot
PerplexityBot
PiplBot
TurnitinBot
UptimeRobot


facebookexternalhit meta-externalads * PetalBot Scrapy uptimerobot viberbot YaK Yandex Yeti


Googlebot
Googlebot-Image
bingbot
Baiduspider-news
Baiduspider
Baiduspider-render
Baiduspider-image
AdsBot-Google
Sogou web spider
Google-Extended
*


grapeshot
*
Amazonbot
anthropic-ai
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
Diffbot
DuckAssistBot
FacebookBot
GPTBot
Google-Extended
Googlebot
magpie-crawler
meta-externalagent
Meta-ExternalFetcher
OAI-SearchBot
Perplexity-User
PerplexityBot
Timpibot
TurnitinBot
Ai2Bot-Dolma
Claude-SearchBot
Claude-User
omgilibot
TtoYouBot
WebGPTBot
Scope3/2.0 (scope3.com)
AmazonAdBot


* Bytespider CCBot claude-searchbot claude-user Claude-Web ClaudeBot FacebookBot Google-Extended MJ12bot NewsNow Perplexity-ai PerplexityBot PiplBot anthropic-ai cohere-ai ia_archiver magpie-crawler news-please omgili omgilibot meta-externalagent meta-externalfetcher A6-Indexer AhrefsBot Amazon-kendra Amazon-qbusiness amazonbot amzn-searchbot amzn-user Applebot-Extended archive.org_bot AwarioRssBot AwarioSmartBot BLP_bbot claritybot DataForSeoBot Diffbot EasouSpider ecoResearch FriendlyCrawler Genieo ImagesiftBot Meltwater Meta-ExternalAgent peer39_crawler peer39_crawler/1.0 PetalBot psbot R6_CommentReader Scrapy Seekr Spinn3r TurnitinBot WebVac WebZip FontReport


* Baiduspider omgili Slurp rogerbot YandexBot PetalBot


* Yandex AdsBot-Google facebookexternalhit Twitterbot


PetalBot
bedrock-agentcore-browser
Amazon-Bot
Amazonbot
AmazonProductDiscoveryBot
Amzn-SearchBot
Amzn-User
Arquivo-web-crawler
bnf.fr_bot
Bytespider
Claude-Extension
ClaudeBot
Claude-User
Claude-SearchBot
Comet
CloudFlare-AlwaysOnline
Cloudflare-Healthchecks
CloudflareBrowserRendering
Cloudflare-Radar
CCBot
DeepSeekBot
DuckAssistBot
archive.org_bot
ia_archiver
MistralAI-User
Moreover
Adbeat
Argus
Baiduspider
Baiduspider-image
Bitdefender
Blogtrottr
BloombergBot
BoardReader
checkmy.ws
Clickagy
CookieBot
cXensebot
DataObserver
DomainStatsBot
Embedly
Feedspotbot
FlipboardProxy
Getintent
Google-Apps-Script
GoogleDrive
Grapevine6
Gumgum
HaloBot
HubSpot Crawler
IbouBot
Inout
LinerBot
MJ12bot
Manus
Meltwater
Metajob
Muckrack
NetSeer
Netvibes
Onyphe
OpenClaw
Opoint
OutbrainBot
PlumAnalytics
ProSitemaps
PerplexityBot
Perplexity-User
proximic
PubMatic
Quantcastbot
SearchConcepts
Semantic-Visions
SentiBot
Sindup
Sirdata
SmartologyBot
Superfeedr
surlybot
Synaptive
TinEye
Trendiction
TurnitinBot
Ubermetrics
Vebidoo
Webcorp
Weborama
WebPageTest
YandexAccessibilityBot
YandexDirect
YandexDirectDyn
YaDirectFetcher
YandexFavicons
YandexMetrika
YandexMobileBot
YandexNews
YandexRCA
YandexRenderResourcesBot
YandexUserproxy
*


CrazyWebCrawler-Spider
*
pimeyes.com crawler
GPTBot
GPTUser
ClaudeBot
Claude-User
PerplexityBot
Perplexity-User


Yandex Googlebot GPTBot Shodan Anews/1.0 *


ChatGPT-User OAI-SearchBot PerplexityBot ClaudeBot GoogleOther GPTBot Google-Extended BingPreview Googlebot Googlebot-image * Baiduspider Yandex Mediapartners-Google AdsBot-Google AdsBot-Google-Mobile


trovitBot
Exabot
sogou spider
Sosospider
peer39_crawler/1.0
AdsBot-Google
AdsBot-Google-Mobile
google-amphtml
AmazonAdBot
FireCrawlAgent
bingbot
*


AI2Bot
Applebot-Extended
Bytespider
CCBot
ClaudeBot
Diffbot
cohere-training-data-crawler
Cotoyogi
Datenbank Crawler
FacebookBot
Factset_spyderbot
Google-Extended
GPTBot
ICC-Crawler
Kangaroo Bot
Meta-ExternalAgent
netEstate Imprint Crawler
omgili
PanguBot
PetalBot
SemrushBot-OCOB
Timpibot
Webzio-Extended
Algolia Crawler
*


* CCBot GPTBot ClaudeBot MistralAI-User Google-Extended PerplexityBot LinkedInBot Twitterbot meta-externalagent OAI-SearchBot ChatGPT-User


*
008
AddThis.com
admantx
BDCbot
Bender
BIXOCRAWLER
bl.uk_lddc_bot
BLEXBot
BUbiNG
Cliqzbot
CNCDialer
crawler4j
CrystalSemanticsBot
CyberAlert
DigExt
discobot
discoverybot
dloader
dloader(NaverRobot)
DOC
dotbot
Download Ninja
DTS Agent
Exabot
Ezooms
FairShare
Fetch
Flamingo_SearchEngine
Genieo
Gigabot
grub-client
Heritrix
heritrix/3.3.0
HTTrack
ia_archiver
integromedb
IstellaBot
JikeSpider
Jyxobot
k2spider
Kimengi
Kimengi/nineconnections.com
larbin
LexxeBot/1.0
libwww
linko
Livelapbot
magpie-crawler
Maxthon
MetaURI
Microsoft.URL.Control
MJ12bot
Moreover
Moreoverbot
MSIECrawler
nabot
NaverBot
NerdByNature.Bot
netEstate NE Crawler
NetSeer crawler
Newscan
NextGenSearchBot
NPBot
nutch
Offline Explorer
omgilibot
Orthogaffe
PiplBot
Pixray-Seeker
proximic
psbot
QuerySeekerSpider
rogerbot
seokicks
SEOkicks-Robot
SiteBot
SiteBot/0.1
sitecheck.internetseer.com
SiteSnagger
Slurp
sogou
Sosospider
spbot
Spinn3r
Teleport
TeleportPro
trendictionbot
trovitBot
TurnitinBot
UbiCrawler
uMBot-LN
UnisterBot
UniversalFeedParser
WBSearchBot
WebCopier
WebReaper
WebStripper
WebZIP
WeSEE:Search


* Googlebot-Image ia_archiver SemrushBot MJ12bot AhrefsBot


* Googlebot LinkedInBot CCBot Google-Extended ia_archiver omgili omgilibot


*
008
Linguee
mxbot
ip-web-crawler.com
Screaming Frog SEO Spider


* Googlebot Googlebot-Image AdsBot-Google-Mobile AdsBot-Google ia_archiver archive.org_bot ia_archiver-web.archive.org AhrefsBot


*
Baiduspider
BaiduMobaider
BaiduImagespider
bingbot
Yeti
psbot
BecomeBot
Teoma
Ask Jeeves
ListRan
GPTBot
grapeshot
YouBot
Leikibot
Sogou
Google-Extended
Mediapartners-Google
Googlebot
AmazonAdBot


GPTBot ChatGPT-User OAI-SearchBot ClaudeBot anthropic-ai PerplexityBot Google-Extended Applebot-Extended CCBot Bytespider Amazonbot *


*
Rogerbot
Exabot
MJ12bot
Dotbot
Gigabot
AhrefsBot
BlackWidow
ChinaClaw
Custo
DISCo
Download\ Demon
eCatch
EirGrabber
EmailSiphon
EmailWolf
Express\ WebPictures
ExtractorPro
EyeNetIE
FlashGet
GetRight
GetWeb!
Go!Zilla
Go-Ahead-Got-It
GrabNet
Grafula
HMView
HTTrack
Image\ Stripper
Image\ Sucker
Indy\ Library
InterGET
Internet\ Ninja
JetCar
JOC\ Web\ Spider
larbin
LeechFTP
Mass\ Downloader
MIDown\ tool
Mister\ PiX
Navroad
NearSite
NetAnts
NetSpider
Net\ Vampire
NetZIP
Octopus
Offline\ Explorer
Offline\ Navigator
PageGrabber
Papa\ Foto
pavuk
pcBrowser
RealDownload
ReGet
SiteSnagger
SemrushBot
SmartDownload
SuperBot
SuperHTTP
Surfbot
tAkeOut
Teleport\ Pro
VoidEYE
Web\ Image\ Collector
Web\ Sucker
WebAuto
WebCopier
WebFetch
WebGo\ IS
WebLeacher
WebReaper
WebSauger
Website\ eXtractor
Website\ Quester
WebStripper
WebWhacker
WebZIP
Wget
Widow
WWWOFFLE
Xaldon\ WebSpider
Zeus


Amazonbot PerplexityBot Perplexity-User ClaudeBot Claude-User GPTBot ChatGPT-User FacebookExternalHit FacebookBot Twitterbot LinkedInBot Bingbot *


Mediapartners-Google Googlebot Googlebot-Image Googlebot-News Googlebot-Video GoogleOther Adsbot-Google Storebot-Google AdsBot-Google-Mobile APIs-Google Twitterbot Applebot Ouestfrancebot Taboolabot Proximic upday Bingbot exabot Slurp ia_archiver grapeshot facebookexternalhit Qwantbot cXensebot AmazonAdBot IbouBot ChatGPT-User Claude-User Claude-SearchBot MistralAI-User PerplexityBot Perplexity-User DuckAssistBot Gemini-DeepResearch Applebot-Extended ClaudeBot Google-Extended Google-NotebookLM FacebookBot Amazonbot Bytespider OAI-SearchBot Meta-ExternalAgent Meta-ExternalFetcher Meta-WebIndexer 5emeRue 5erue adequat adequat-systems AgentCore-Browser Ai2Bot-Dolma AI2Bot/v2.0 Aihitdata.com AmazonBot AmiSoftware amazon-kendra anthropic-ai Argus Asknread asknread.com Augure auramundi bedrockbot/3.0 Bloodhound CCBot ChatGPT-Agent ChatGLM-Spider Channel3Bot Cision Claude-Web coexel Cohere-ai ConveraCrawler Corporama CotoyogiBot cydralspider DeepSeekBot Diffbot Digimind DownloadNinja downloadexpress EDD ellisphere eureka Europresse Explore Factiva Fasterfox Fetch gammaSpider GoogleAgent-Shopping Google-CloudVertextBot GPTBot grub-client HTTrack IAskBot ia_archiver-web.archive.org ICC-Crawler ImageSiftBot img2dataset


*
Googlebot-News
ChatGPT-User
OAI-SearchBot
Claude-User
Claude-Web
Claude-SearchBot
Perplexity-User
Google-Extended
Gemini-Deep-Research
MistralAI-User
GPTBot
Operator
ClaudeBot
anthropic-ai
GoogleOther
Google-CloudVertexBot
GoogleOther-Image
GoogleOther-Video
Applebot-Extended
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
Amazonbot
bedrockbot
NovaAct
PerplexityBot
cohere-ai
cohere-training-data-crawler
Bytespider
TikTokSpider
CCBot
Diffbot
omgili
omgilibot
Webzio-Extended
YouBot
PetalBot
PanguBot
SemrushBot-OCOB
SemrushBot-SWA
ImagesiftBot
img2dataset
FirecrawlAgent
ShapBot
Thinkbot
VelenPublicWebCrawler
Andibot
PhindBot
YandexAdditional
YandexAdditionalBot
FriendlyCrawler
Crawlspace
ICC-Crawler
SBIntuitionsBot
Panscient
panscient.com
Timpibot
AddSearchBot
aiHitBot
iaskspider
LinerBot
CloudVertexBot
Google-Firebase
Terrecotta
TerraCotta
Kangaroo Bot
meta-webindexer
Brightbot
Brightbot 1.0
SearchmetricsBot
deepcrawl
rogerbot
MJ12bot
OrangeBot
OrangeBot-Collector
Slurp
Adixbot
YandexBot
AhrefsBot
Sistrix
SemrushBot
Screaming Frog SEO Spider


* Mediapartners-Google AppEngine-Google Yandex AhrefsBot bingbot slurp Seznambot DataForSeoBot dotbot grapeshot ChatGPT-User ImagesiftBot OAI-SearchBot MJ12bot Amazonbot ClaudeBot meta-externalagent CCBot PerplexityBot Diffbot anthropic-ai YouBot FeedBurner


* MachineLearning GPTBot Google-Extended PerplexityBot Amazonbot ClaudeBot Omgilibot FacebookBot Applebot anthropic-ai Bytespider Claude-Web Diffbot ImagesiftBot Omgili YouBot


Mozilla/5.0 (compatible; parse.ly scraper/0.16; +http://parsely.com)


* AhrefsBot CCBot GPTBot Yandexbot AmazonAdBot


Googlebot * ClaudeBot CCBot GPTBot Google-Extended


* Mediapartners-Google Adsbot-Google Twitterbot


* Baiduspider Twitterbot IndeedBot Jobblebot neuvoo GPTBot OAI-SearchBot ChatGPT-User


sosospider
yandex bot
larbin
BizwikiBot
ScoutJet
Riddlerbot
SemrushBot
ShopWiki
ia_archiver
*


* Googlebot CCBot GPTBot ChatGPT-User Slurp


* Yandex YandexBot YandexMobileBot YandexDirect APIs-Google AdsBot-Google-Mobile AdsBot-Google Mediapartners-Google Amazonbot PetalBot Bytespider anthropic-ai CCBot Baiduspider-render Baiduspider OAI-SearchBot GPTBot ChatGPT-User/2.0 Google-Extended PerplexityBot ClaudeBot Claude-SearchBot


*
AddSearchBot
AI2Bot
Amazonbot
anthropic-ai
Applebot
Applebot-Extended
bigsur.ai
Bytespider
CCBot
ChatGPT Agent
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CloudVertexBot
cohere-ai
cohere-training-data-crawler
Cotoyogi
Datenbank Crawler
Devin
Diffbot
DuckAssistBot
FacebookBot
Gemini-Deep-Research
Google-Extended
GoogleAgent-Mariner
GoogleOther
Googlebot
Bingbot
GPTBot
ICC-Crawler
Kangaroo Bot
LinerBot
meta-externalagent
MistralAI-User
netEstate Imprint Crawler
NovaAct
OAI-SearchBot
omgili
PanguBot
Perplexity-User
PerplexityBot
PetalBot
QualifiedBot
Timpibot
Webzio-Extended
YouBot


*
SmartViper
Mediapartners-Google
CCBot
Omgilibot
Omgili
FacebookBot
Meta-ExternalAgent
Meta-ExternalFetcher
Diffbot
DuckAssistBot
AI2Bot
Bytespider
ImagesiftBot
Kangaroo Bot
cohere-ai
Timpibot
Webzio-Extended
YouBot
AmazonAdBot


grapeshot
*
Amazonbot
anthropic-ai
Applebot-Extended
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
cohere-ai
Diffbot
DuckAssistBot
FacebookBot
GPTBot
Google-Extended
Googlebot
magpie-crawler
meta-externalagent
Meta-ExternalFetcher
OAI-SearchBot
Perplexity-User
PerplexityBot
Timpibot
TurnitinBot
Ai2Bot-Dolma
Claude-SearchBot
Claude-User
omgilibot
TtoYouBot
WebGPTBot
Scope3/2.0 (scope3.com)
AmazonAdBot


*
OAI-SearchBot
Googlebot-News
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot-Extended
Bytespider
CCBot
Claude-Web
ClaudeBot
Claude-SearchBot
cohere-ai
cohere-training-data-crawler
Crawlspace
Diffbot
FacebookBot
FriendlyCrawler
Google-Extended
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
Kangaroo Bot
Meta-ExternalAgent
omgili
omgilibot
PanguBot
PerplexityBot
PetalBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot


*
OAI-SearchBot
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot-Extended
Bytespider
CCBot
Claude-Web
ClaudeBot
Claude-SearchBot
cohere-ai
cohere-training-data-crawler
Crawlspace
Diffbot
FacebookBot
FriendlyCrawler
Google-Extended
iaskspider/2.0
ICC-Crawler
ImagesiftBot
img2dataset
Kangaroo Bot
Meta-ExternalAgent
omgili
omgilibot
PanguBot
PerplexityBot
PetalBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetrade indexer bot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot


*
AdIdxbot
Facebookbot
Facebot
ClaudeBot
CriteoBot/0.1
alexabot
ia_archiver
alexa site audit
Bytespider
omgili
Dataprovider.com
dcrawl
Nutch
HTTrack
HTTrack 3.0
MetaInspector
Offline Explorer
008
Slurp
SeznamBot
SearchmetricsBot
Feedonomics
EtaoSpider
BLEXBot
ImagesiftBot
trendictionbot
EyeMonIT Uptime Bot
AhrefsSiteAudit
Mail.RU_Bot
AhrefsBot
Baiduspider
SputnikBot
MJ12bot
FeedBurner
Exabot
proximic
Scrapy
SemrushBot
coccocbot
IAS Crawler
dotbot
ltx71
Sogou web spider
Pingdom.com_bot
GetIntent Crawler
expo9
PetalBot
Yandex
bingbot
Mediapartners-Google
Adsbot-Google
OAI-SearchBot
GPTBot
ChatGPT-User
Google-Extended
meta-externalagent


*
FairShare
MetaURI
nabot
NaverBot
Kimengi/nineconnections.com
Kimengi
DigExt
DTS Agent
dloader
dloader(NaverRobot)
magpie-crawler
MJ12bot
AhrefsBot
Exabot
QuerySeekerSpider
UniversalFeedParser
Flamingo_SearchEngine
Livelapbot
Wotbot
Wotbox
BIXOCRAWLER
proximic
Genieo
Heritrix
rogerbot
spbot
trendictionbot
uMBot-LN
SEOkicks-Robot
nutch
crawler4j
dotbot
Moreover
omgilibot
Cliqzbot
BUbiNG
PiplBot
Gigabot
GPTBot
ChatGPT-User
CCBot
anthropic-ai
ClaudeBot
PerplexityBot
Google-Extended
Applebot-Extended
Meta-ExternalAgent
Meta-ExternalFetcher
Baiduspider
AI-SearchBot
cohere-ai
OAI-SearchBot
Bytespider
Amazonbot
Diffbot
YouBot
PetalBot
Timpibot
DataForSeoBot
omgili


GPTBot ClaudeBot anthropic-ai Claude-Web Google-Extended Applebot-Extended CCBot Bytespider Amazonbot Meta-ExternalAgent cohere-ai PerplexityBot Baiduspider Baiduspider-video Baiduspider-image *


* grapeshot GPTBot ClaudeBot Google-Extended Amazonbot CCBot


*
AI2Bot
Adequat
Alexibot
AlvinetSpider
AmiSoftware
anthropic-ai
Applebot-Extended
Argus
AudigentAdBot
Augure
BeOpBot
BizInformation
Black Hole
BunnySlippers
Bytespider
CCBot
Cision
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Coexel
cohere-ai
cohere-training-data-crawler
Corporama
DataForSeoBot
Diffbot
Digimind
dotbot
Download Ninja
DuckAssistBot
Ellisphere
EmailCollector
EmailSiphon
EmailWolf
FacebookBot
Factiva
Fetch
GPTBot
Gemini-Deep-Research
Google-Extended
HTTrack
Jetbot
Kangaroo Bot
kbcrawl
linkfluence
lwp-trivial
Manageo
Meltwater
Mention
Meta-ExternalAgent
Meta-ExternalFetcher
MistralAI-User
moget
Moreover
Naverbot
NetAnts
NetAttache
NetMechanic
NiceRsPRO
Offline Explorer
Onclusive
PanguBot
Perplexity-User
PerplexityBot
Proxem
psbot
Qwam content intelligence
Raven
scoop.it
Score3
SemrushBot
Sindup
SiteSucker
Sogou web spider
Spotter
SuperBot
Synthesio
Talkwalker
Teleport
TeleportPro
Trendeo
Trendybuzz
True_Robot
UptimeRobot
UrlPouls
VCI
Vecteurplus
VerticalSearch
WebBandit
WebCopier
WebReaper
WebSauger
WebStripper
WebZIP
Webzio-Extended
Wget
WinHTTrack
Winello


*
AliyunSecBot
AmazonBot
anthropic-ai
Applebot-Extended
archive.org_bot
AudigentAdBot
AwarioRssBot
AwarioSmartBot
BLEXBot
Bytespider
CCBot
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
Claude-Web
cohere-ai
DataForSeoBot
Diffbot
DuckAssistBot
EchoboxBot
FriendlyCrawler
Baiduspider
Bingbot
Slurp
GPTBot
ImagesiftBot
Jetslide
magpie-crawler
MyCentralAIScraperBot
NewsNow
news-please
OAI-SearchBot
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
PerplexityBot
Perplexity-User
Poseidon Research Crawler
quillbot.com
Quora-Bot
Scrapy
SeekrBot
SeznamHomepageCrawler
TaraGroup Intelligent Bot
Timpibot
TurnitinBot
ViennaTinyBot
YouBot
Google-Extended
Google-CloudVertexBot
facebookexternalhit


* Amazonbot Applebot-Extended Bytespider CCBot ClaudeBot CloudflareBrowserRenderingCrawler Google-Extended GPTBot meta-externalagent


* Googlebot-News AhrefsBot Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ChatGPT-User Claude Claude-Web ClaudeBot coccocbot-web cohere-ai DeepSeekBot Diffbot Exabot Google-Extended GPTBot Meta-ExternalFetcher MistralAI-User MistralAI-User/1.0 MJ12bot OAI-SearchBot Perplexity-User PerplexityBot PetalBot proximic SemrushBot SemrushBot-OCOB SemrushBot-SWA Timpibot Yandex YandexAdditional YandexAdditionalBot YandexBot YandexImages YandexRenderBot YouBot


* Googlebot Googlebot-Image MSNBot Slurp FlipboardProxy iSec_Bot TurnitinBot ICC-Crawler trendkite-akashic-crawler TinEye-bot R6_CommentReader BLEXBot Nutch ChatGPT-User ClaudeBot Claude-Web Timpibot omgili cohere-ai anthropic-ai Bytespider CCBot Diffbot Meta-ExternalAgent Google-Extended GPTBot PerplexityBot Omgilibot Webzio-Extended YouBot


* Twitterbot GPTBot ChatGPT-User CCBot anthropic-ai Amazonbot Applebot-Extended Bytespider ClaudeBot Claude-Web cohere-ai Diffbot Google-Extended Meta-ExternalFetcher PerplexityBot YouBot DeepSeekBot MistralAI-User MistralAI-User/1.0 PetalBot Timpibot YandexAdditional YandexAdditionalBot YandexBot YandexRenderBot Perplexity-User coccocbot-web OAI-SearchBot Claude


Googlebot
Googlebot-Mobile
Yahoo! Slurp
Slurp
slurp
msnbot
bingbot
ia_archiver
AdsBot-Google
AdsBot-Google-Mobile
Mediapartners-Google
proximic
Google-Extended
OAI-SearchBot
ChatGPT-User
facebookexternalhit


* Yandex AdsBot-Google facebookexternalhit Twitterbot


AhrefsBot
AI2Bot
Ai2Bot-Dolma
Amazonbot
anthropic-ai
Applebot-Extended
BLEXBot
BoardReader
Bytespider
CCBot
ClaudeBot
cohere-ai
cohere-training-data-crawler
DeepSeek
DeepSeekBot
Diffbot
dotbot
FacebookBot
Google-Extended
GPTBot
ia_archiver
ICCrawler
ImagesiftBot
Linguee Bot
Meta-ExternalAgent
MJ12bot
Omgili
Omgilibot
PerplexityBot
PetalBot
SemrushBot
Sogou inst spider
Sogou web spider
The Knowledge AI
trendictionbot
TurnitinBot
Tweetmemebot
YouBot
webzio
Webzio-Extended
AmazonAdBot
facebookexternalhit
ChatGPT-User
Claude-User
Claude-SearchBot
OAI-SearchBot
Perplexity-User
Mediapartners-Google
Twitterbot
Googlebot
Yeti
Daum
ZumBot
bingbot
Applebot
DuckDuckBot
*


* AmazonAdBot GPTBot MJ12bot Mediapartners-Google SpriftCrawler TrovitBot cbBot oodlebot spider CCbot


*
EtaoSpider
GPTBot
CCBot
PerplexityBot
Google-Extended
GoogleAgent-Mariner
GoogleAgent-Shopping
ClaudeBot
meta-externalagent
Bytespider
Scrapy
PetalBot
Devin
omgili
AI2Bot
Gemini-Deep-Research
PanguBot
MistralAI-User
Diffbot
Sidetrade indexer bot
Google-NotebookLM
DuckAssistBot
Copilot
Ai2Bot-Dolma
Google-CloudVertexBot
meta-externalfetcher
iaskspider
wpbot
aiHitBot
cohere-ai
FriendlyCrawler
img2dataset
VelenPublicWebCrawler
YouBot
Brightbot
Claude-User
Claude-SearchBot
Perplexity-User
ISSCyberRiskCrawler
CopilotNative
CopilotSapphire
imgproxy
SemrushBot-SWA
ChatGPT-User
OAI-SearchBot
Timpibot
Thinkbot
AddSearchBot
AI2Bot-DeepResearchEval
Andibot
Anomura
atlassian-bot
BuddyBot
Channel3Bot
ChatGLM-Spider
Claude-Web
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Datenbank Crawler
DeepSeekBot
Echobot Bot
EchoboxBot
factset_spyderbot
Google-Firebase
GoogleOther
Grok-DeepSearch
GrokBot
iAskBot
IbouBot
ICC-Crawler
ImagesiftBot
imageSpider
Kangaroo Bot
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
Linguee Bot
LinkupBot
Manus-User
meta-webindexer
netEstate Imprint Crawler
Panscient
panscient.com
PhindBot
Poggio-Citations
QualifiedBot
SBIntuitionsBot
ShapBot
TavilyBot
TerraCotta
TwinAgent
webzio-extended
WRTNBot
xAI-Grok
YaK
ZanistaBot


* barkrowler SiteAuditBot PetalBot Bytespider ChatGPT-User


psbot NPBot-1/2.0 NPBot Baiduspider Grok Google-CloudVertexBot AmazonAdBot GPTBot CCBot Google-Extended Applebot-Extended ClaudeBot Amazonbot Meta-ExternalAgent *


Googlebot
Baiduspider
bingbot
Sogou web spider
Sogou wap spider
YisouSpider
BaiduSpider-ads
*


* Googlebot Bingbot AhrefsBot Ai2Bot Ai2Bot-Dolma Amazonbot anthropic-ai Applebot Applebot-Extended AwarioRssBot AwarioSmartBot Bytespider CCBot ChatGPT-User ClaudeBot Claude-Web cohere-ai DataForSeoBot Diffbot dotbot FacebookBot FriendlyCrawler Google-Extended GPTBot ia_archiver ImagesiftBot img2dataset magpie-crawler Meta-ExternalAgent Meta-ExternalFetcher NewsNow MJ12bot news-please OAI-SearchBot omgili omgilibot peer39_crawler peer39_crawler/1.0 PerplexityBot PetalBot Scrapy SemrushBot SirdataBot Timpibot TurnitinBot VelenPublicWebCrawler Webzio-Extended Yeti YouBot Twitterbot facebookexternalhit


*
Amazonbot
AlibabaBot
Ai2Bot-Dolma
Applebot-Extended
Bytespider
CCBot
ChatGLM-Spider
ClaudeBot
Claude-Web
CloudVertexBot
cohere-training-data-crawler
Cotoyogi
Datenbank Crawler
Diffbot
FacebookBot
Google-Extended
GoogleOther
GPTBot
ICC-Crawler
imageSpider
Kangaroo Bot
laion-huggingface-processor
LCC
meta-externalagent
netEstate Imprint Crawler
omgili
PanguBot
PerplexityBot
SBIntuitionsBot
Spider
Timpibot
VelenPublicWebCrawler
webzio-extended
YouBot
008
Dataprovider.com
dcrawl
HelloworkJobPostingBot
HTTrack
HTTrack 3.0
KrawlerBot
MetaInspector
MetaJobBot
newspaper
Nutch
Offline Explorer
OpenindexSpider
Potions
Scrapy
ServerHunterSpider
StatsDroneBot
CloudflareBrowserRenderingCrawler
aa
AhrefsBot
Attracta
Barkrowler
BrightEdge Crawler
BLEXBot
CaliberBot
Caliperbot
ClarityBot
Clearscopebot
Cloudtrellis
cludo.com
Cocolyzebot
cognitiveSEO Crawler
contentking
Convermax
Cxense
DataForSeoBot
DataForSEO Bot
DomainStatsBot
DotBot
Dragonbot
Huckabot
Huckabuy Bot
hypestat
iaskspider
img2dataset
James BOT
LinkCheck by Siteimprove
LinkChecker Bot
linkchecker.pro
linkdexbot
LinksIndexerBot
MarketGoo
MBCrawler
MegaIndex.ru
MJ12bot
Moz dotbot
Moz rogerbot
Nitro-
NitroBot
online-webceo-bot
Prerender
Pro Sitemaps
Readable
RevvimGort
rogerbot-crawler


* SemrushBot SemrushBot-SA Riddler BLEXBot serpstatbot bingbot Bytespider PetalBot


MJ12bot
Mediapartners-Google*
IsraBot
Orthogaffe
UbiCrawler
DOC
Zao
sitecheck.internetseer.com
Zealbot
MSIECrawler
SiteSnagger
WebStripper
WebCopier
Fetch
Offline Explorer
Teleport
TeleportPro
WebZIP
linko
HTTrack
Microsoft.URL.Control
Xenu
larbin
libwww
ZyBORG
Download Ninja
fast
wget
grub-client
k2spider
NPBot
WebReaper
SemrushBot
*


*
Amazonbot
Applebot-Extended
Bytespider
CCBot
ClaudeBot
CloudflareBrowserRenderingCrawler
Google-Extended
GPTBot
meta-externalagent
OAI-SearchBot
ChatGPT-User
anthropic-ai
Meta-ExternalAgent
Meta-ExternalFetcher
FacebookBot
PerplexityBot
cohere-ai
cohere-training-data-crawler
AI2Bot
Diffbot
DuckAssistBot
Webzio-Extended
Omgilibot
ImagesiftBot
Kangaroo Bot
PanguBot
Timpibot
YouBo
Googlebot
Bingbot
DuckDuckBot
Yandex
Baiduspider
Bravebot
AhrefsBot
AhrefsSiteAudit


* Googlebot-News Applebot Bingbot Amazonbot amazon-kendra bedrockbot Anthropic-ai AwarioRssBot AwarioSmartBot Bytespider CCBot GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-Web Cohere-ai DataForSeoBot FacebookBot meta-externalagent meta-externalfetcher Google-Extended Gemini-Deep-Research ImagesiftBot Magpie-crawler Omgili Omgilibot PerplexityBot Perplexity-User YouBot huggingface NewsNow news-please Diffbot Scrapy TurnitinBot Applebot-Extended AI2Bot MistralAI-User ChatGLM-Spider Cotoyogi DeepSeekBot EchoboxBot iaskspider ICC-Crawler LinerBot PanguBot ProRataInc QualifiedBot SeekrBot VelenPublicWebCrawler WARDBot WPBot WRTNBot ZanistaBot


* BLEXBot AhrefsBot Amazonbot SemrushBot dotbot


* GPTBot ClaudeBot PerplexityBot Google-Extended


msnbot
*
Amazonbot
anthropic-ai
Applebot-Extended
CCBot
ClaudeBot
Claude-Web
cohere-ai
Diffbot
discobot
dotbot
FacebookBot
google-extended
Google-Extended
Googlebot
GPTBot
PerplexityBot
TurnitinBot
magpie-crawler
meta-externalagent
Meta-ExternalFetcher
Bytespider
DuckAssistBot
Perplexity-User
Timpibot
Ai2Bot-Dolma
omgilibot
TtoYouBot
WebGPTBot
Slurp
Verity/1.1
Scope3/2.0 (scope3.com)
AmazonAdBot
yacybot


* bingbot AdsBot-Google AdIdxBot Twitterbot Bytespider CCBot ChatGLM-Spider ClaudeBot PerplexityBot anthropic-ai AmazonBot meta-externalagent GPTBot Applebot-Extended


Mediapartners-Google ia_archiver AdsBot-Google *


* OAI-SearchBot ChatGPT-User ChatGPT-User/2.0 anthropic-ai ClaudeBot claude-web PerplexityBot Perplexity-User Google-Extended Googlebot Bingbot Amazonbot Applebot GPTBot Baiduspider Yandex ia_archiver


* Googlebot-Mobile Google-Extended Googlebot bingbot msnbot Crowsnest Twitterbot