📘 This use case demonstrates breaking server-log activity down by bot family so search engine crawlers and AI crawlers can be evaluated and managed separately.
Best for: Teams managing crawl budget and bot access · Uses: LogAnalyzer, via Claude or ChatGPT
Overview
Search engine crawlers, such as Googlebot and Bingbot, and AI crawlers, such as GPTBot, hit your site for different reasons: one is building a search index, the other is gathering content to train or ground an AI model. Botify MCP can break your log data down by bot family so you can evaluate each on its own terms.
Prerequisites
Botify MCP connected to your AI assistant, such as Claude or ChatGPT. See Connecting to the Botify MCP for setup steps.
LogAnalyzer enabled and ingesting logs for the project.
Description of the problem
Treating all bot traffic the same wastes crawl budget and hides what each crawler family is actually doing on the site, which matters when deciding how to set crawl or serving rules for search engine crawlers versus AI crawlers.
Application
Ask your AI assistant to break down hit volume by bot family from your server logs:
Show Googlebot, Bingbot, and GPTBot hits for [project] over the last 30 days, separated from human referral visits.
Why this prompt works:Naming the bot families, rather than asking for "bot traffic," pulls each one from its own row in the log data instead of collapsing them into one undifferentiated count.
A stated window (30 days) matches how LogAnalyzer buckets crawl activity, so the numbers reconcile with what you'd see filtering the same range in the UI.
Separating crawler hits from human referral visits keeps crawl volume from diluting into total traffic — the comparison that actually matters for a crawl-budget decision.
LogAnalyzer's own UI groups AI bots as "OpenAI Bots" and "Other AI Bots." Naming individual bots (GPTBot, ClaudeBot) still resolves correctly since those are the user agents behind each group — ask for the group names directly only if you need strict parity with a saved LogAnalyzer report.
Compare crawl-to-visit ratios and flag wasted requests, such as repeated 403s or 404s.
Use the breakdown to decide where to adjust crawl or serving rules for search engine crawlers versus AI crawlers.
Follow-up prompts
"Break GPTBot and ClaudeBot out separately instead of grouping them as AI crawlers." — useful once you're deciding per-bot rules rather than one blanket AI-crawler policy.
"Show the same breakdown for the last 90 days so I can see whether GPTBot's share is growing." — turns a one-time snapshot into a trend. For an ongoing version of this, see Turning a Bot Traffic Breakdown into a Recurring Report.
"Flag any URLs where search engine crawler hits started returning 403s or 404s in the last 30 days." — connects the traffic split back to a specific serving-rule change or rollout.
Make it repeatable
Ask your assistant to run this on a schedule instead of re-typing it: "Run this every Monday for [project]."
Save the exact prompt wording your team settles on — bot names, window, project — somewhere shared. Otherwise the next person to ask this question phrases it slightly differently and gets numbers that don't reconcile with yours.
Result
Because search engine crawlers and AI crawlers are tracked separately in the underlying log data, you get a clear read on each without cross-referencing spreadsheets manually.
Read next
Related topics