🛠️ This lists the bots, or web crawlers, supported in Botify.

Overview

Activity for the following bots is available in LogAnalyzer reports when included in your log files. The activity for the bots indexing content for search engines is also included in SiteCrawler reports. Activity from other bots is excluded in SiteCrawler since these bots only access and crawl web pages to gather information or perform analysis and, therefore, do not impact your SEO. Including the crawl activity for these non-indexing bots would skew your analyses with data that does not affect your site evaluation.

Bot List

If you filter your log data, update your filters to include these bot user agents.

Source/Owner	Bot Name	In SiteCrawler?
Google	AdsBot Desktop	✅
	AdsBot Mobile	✅
	Googlebot Desktop	✅
	Googlebot Images	✅
	Google InspectionTool	`❌`
	Google Video	✅
	Google Other	`❌`
	Google AdSense Desktop	✅
	Google AdSense Mobile	✅
	Googlebot Smartphone	✅
	Google Storebot	✅
	Google Video	✅
Bing	Bingbot	✅
OpenAI	GPTBot	`❌`
	ChatGPT-User	`❌`
	OAI-SearchBot	✅
AI Bots	AmazonBot	✅
Anthropic	Anthropic-ai	`❌`
ByteDance/TikTok	Bytespider	`❌`
Common Crawl	CCBot	`❌`
Anthropic	ClaudeBot	`❌`
Anthropic	Claude-Web	`❌`
Meta	FacebookBot	`❌`
Meta	Meta-ExternalAgent	`❌`
Mistral	MistralAI-User	`❌`
Perplexity AI	PerplexityBot	✅
	Perplexity-User	`❌`
You.com	YouBot	✅
Apple	Applebot	✅
Yandex	Yandex Antivirus	✅
	Yandex Blogs	✅
	Yandex Catalog	✅
	Yandex Direct	✅
	Yandex Favicons	✅
	Yandex ImageResizer	✅
	Yandex Images	✅
	Yandex Market	✅
	Yandex Media	✅
	Yandex Metrika	✅
	Yandex News	✅
	Yandex Pagechecker	✅
	Yandex Search	✅
	Yandex Video	✅
	Yandex Webmaster	✅
	Yandex Zakladki	✅
Naver	Naver Search	✅
Baidu	Baidu Bookmark	✅
	Baidu Business (Ads)	✅
	Baidu Image	✅
	Baidu Mobile	✅
	Baidu News	✅
	Baidu Search	✅
	Baidu Union	✅
	Baidu Video	✅

Supported Bots in SpeedWorkers

These bots are also supported in SpeedWorkers; however, you must have Bing in your contract to serve to AI bots through SpeedWorkers.

User Agents

To identify bots in logs or when configuring CDN settings, include the following user agents according to those allowed by your contract:

Botify:

*botify-bot-sw/*

Google:

*googlebot/*

*googleinspection/*

*googleother/*

Bing:

*bingbot/*

Other Search bots:

*yandexbot/*

*yandexmobilebot/*

*baiduspider/*

*baiduspider+/*

*applebot/*

*yeti/*

AI bots:

*GPTBot/*

*Amazonbot/*

*Anthropic/*

*Bytespider/*

*CCBot/*

*ChatGPT-User/*

*Claudebot/*

*Claude/*

*Facebook/*

*Meta-External/*

*OAI-SearchBot/*

*Perplexity/*

*YouBot/*

💡 When identifying user agents to intercept for SpeedWorkers, use the format required by your CDN, such as:

GPTBot|Amazonbot|Anthropic|Bytespider|CCBot|ChatGPT-User|Claudebot|Claude|Facebook|Meta-External|OAI-SearchBot|Perplexity|YouBot

*GPTBot* *Amazonbot* *Anthropic* *Bytespider* *CCBot* *ChatGPT-User* *Claudebot* *Claude* *Facebook* *Meta-External* *OAI-SearchBot* *Perplexity* *YouBot*

About Bot Authentication

Botify can only authenticate bots that share their IP addresses. The following supported bots do not share their IP addresses. While crawls from these bots will appear in Botify when included in your log data, we cannot validate crawls from them:

Anthropic-ai
Bytespider
CCBot
ClaudeBot
Claude-Web