📘 This lists the bots, or web crawlers, supported in Botify.
Overview
Activity for the following bots is available in LogAnalyzer reports when included in your log files. The activity for the bots indexing content for search engines is also included in SiteCrawler reports. Activity from other bots is excluded in SiteCrawler since these bots only access and crawl web pages to gather information or perform analysis and, therefore, do not impact your SEO. Including the crawl activity for these non-indexing bots would skew your analyses with data that does not affect your site evaluation.
Bot List
If you filter your log data, update your filters to include these bot user agents.
If you filter your log data, update your filters to include these bot user agents.
Source/Owner | Bot Name | In SiteCrawler? |
AdsBot Desktop | ✔️ | |
| AdsBot Mobile | ✔️ |
| Google Agent | ✔️ |
| Googlebot Desktop | ✔️ |
| Googlebot Images | ✔️ |
| Google InspectionTool | ❌ |
| Google Video | ✔️ |
| Google Other | ❌ |
| Google AdSense Desktop | ✔️ |
| Google AdSense Mobile | ✔️ |
| Googlebot Smartphone | ✔️ |
| Google Storebot | ✔️ |
| Google Video | ✔️ |
Bing | Bingbot | ✔️ |
OpenAI | GPTBot | ❌ |
| ChatGPT-User | ❌ |
| OAI-SearchBot | ✔️ |
AI Bots | AmazonBot | ✔️ |
Anthropic | Anthropic-ai | ❌ |
ByteDance/TikTok | Bytespider | ❌ |
Common Crawl | CCBot | ❌ |
Anthropic | ClaudeBot | ❌ |
Anthropic | Claude-Web | ❌ |
Meta | FacebookBot | ❌ |
Meta | Meta-ExternalAgent | ❌ |
Mistral | MistralAI-User | ❌ |
Perplexity AI | PerplexityBot | ✔️ |
| Perplexity-User | ❌ |
You.com | YouBot | ✔️ |
Apple | Applebot | ✔️ |
Yandex | Yandex Antivirus | ✔️ |
| Yandex Blogs | ✔️ |
| Yandex Catalog | ✔️ |
| Yandex Direct | ✔️ |
| Yandex Favicons | ✔️ |
| Yandex ImageResizer | ✔️ |
| Yandex Images | ✔️ |
| Yandex Market | ✔️ |
| Yandex Media | ✔️ |
| Yandex Metrika | ✔️ |
| Yandex News | ✔️ |
| Yandex Pagechecker | ✔️ |
| Yandex Search | ✔️ |
| Yandex Video | ✔️ |
| Yandex Webmaster | ✔️ |
| Yandex Zakladki | ✔️ |
Naver | Naver Search | ✔️ |
Baidu | Baidu Bookmark | ✔️ |
| Baidu Business (Ads) | ✔️ |
| Baidu Image | ✔️ |
| Baidu Mobile | ✔️ |
| Baidu News | ✔️ |
| Baidu Search | ✔️ |
| Baidu Union | ✔️ |
| Baidu Video | ✔️ |
Supported Bots in SpeedWorkers
These bots are also supported in SpeedWorkers; however, you must have Bing in your contract to serve to AI bots through SpeedWorkers.
💡User agent names are case-insensitive in routing rules.
User Agents
To identify bots in logs or when configuring CDN settings, include the following user agents according to those allowed by your contract.
👀 Wildcards are used in this list because some full user agent strings are not publicly available, and new ones are continuously added.
Botify:
botify-bot-sw-test
Google:
*google-agent*
*googlebot/*
*google-inspectiontool/*
*googleother/*
Bing:
*bingbot/*
Other Search bots:
*yandexbot/*
*yandexmobilebot/*
*baiduspider/*
*baiduspider+/*
*applebot/*
*yeti/*
AI bots:
*GPTBot/*
*Amazonbot/*
*anthropic-ai*
*Bytespider*
*CCBot/*
*ChatGPT-User/*
*Claudebot/*
*Claude-Web*
*Claude-User/*
*Claude-Searchbot/*
*FacebookBot/*
*meta-externalagent/*
*OAI-SearchBot/*
*PerplexityBot/*
*Perplexity-User/*
*YouBot/*
👉 When identifying user agents to intercept for SpeedWorkers, use the format required by your CDN, such as:
amazonbot\/|anthropic-ai|applebot\/|bingbot\/|bytespider|ccbot\/|chatgpt-user\/|claude-web|claudebot\/|claude-user\/|claude-searchbot\/|facebookbot\/|google-inspectiontool\/|googlebot\/|googleother|gptbot\/|meta-externalagent\/|oai-searchbot\/|perplexity-user\/|perplexitybot\/|youbot\/|botify-bot-sw-
*amazonbot* *anthropic-ai* *applebot* *bingbot* *bytespider* *ccbot* *chatgpt-user* *claude-web* *claudebot* *claude-user* *claude-searchbot* *facebookbot* *google-inspectiontool* *googlebot* *googleother* *gptbot* *meta-externalagent* *oai-searchbot* *perplexity-user* *perplexitybot* *youbot* *botify-bot-sw-*
amazonbot|anthropic-ai|applebot|bingbot|bytespider|ccbot|chatgpt-user|claude-web|claudebot|claude-user|claude-searchbot|facebookbot|google-inspectiontool|googlebot|googleother|gptbot|meta-externalagent|oai-searchbot|perplexity-user|perplexitybot|youbot|botify-bot-sw-
About Bot Authentication
Botify can only authenticate bots that share their IP addresses. The following supported bots do not share their IP addresses. While crawls from these bots will appear in Botify when included in your log data, we cannot validate crawls from them:
Bytespider
CCBot
ClaudeBot
Claude-Web
See also: