📘 This use case demonstrates running the same bot-family breakdown on a cadence, with the change from the prior period, so crawl activity becomes a tracked trend.
Best for: Teams managing crawl budget and bot access, on an ongoing basis · Uses: LogAnalyzer, via Claude or ChatGPT
Overview
Separating Search Engine and AI Crawler Traffic answers a single question: what is each bot family doing right now. Turning that same breakdown into a recurring report lets your team track how bot activity changes over time, instead of re-asking the question from scratch whenever it comes up.
Prerequisites
Botify MCP connected to your AI assistant, such as Claude or ChatGPT. See Connecting to the Botify MCP for setup steps.
LogAnalyzer enabled and ingesting logs for the project.
A bot traffic breakdown you've already run once. See Separating Search Engine and AI Crawler Traffic.
Description of the problem
Bot activity changes over time. A single breakdown tells you what's true right now, not whether a bot family's share of crawl activity is growing, shrinking, or shifting.
Application
Ask your AI assistant to run the same bot-family breakdown at a set cadence:
Every [week/month], show Googlebot, Bingbot, and GPTBot hits for [project] over the prior period, and note the change from the period before.
Why this prompt works:Asking for the change from the prior period, not just the current count, is what turns a snapshot into a trend.
A set cadence and a stated period keep every run comparable to the last one, the same way Building a Recurring Stakeholder Report keeps a report reproducible from cycle to cycle.
Review the trend, not just the latest snapshot: is a specific bot family's share growing, shrinking, or holding steady.
Feed a meaningful change into whatever decision it affects, such as a serving-rule adjustment or a follow-up check in Checking AI Crawler Access and Brand Citations.
Follow-up prompts
"Which bot family grew the fastest over the last quarter?" — surfaces the trend worth a closer look.
"Flag any period where a bot family's hits dropped by more than [threshold]." — catches a sudden change rather than a gradual one.
Make it repeatable
Ask your assistant to run this on a schedule rather than re-typing it each cycle.
Keep the bot names, project, and cadence wording consistent across runs, the same way you would for any recurring report, so the numbers reconcile period to period.
Result
Bot-family activity becomes a tracked trend your team checks on a cadence, rather than a one-time answer re-derived whenever the question resurfaces.
Read next
Related topics