📘 This article describes the three sources of visit data in Botify and explains when to use each.
Overview
The following Visit data sources can be integrated with Botify:
Logs: Visits from your log files in LogAnalyzer and SiteCrawler.
Analytics: Visits from your analytics provider in SiteCrawler and RealKeywords.
Google Search Console: Clicks from Google Search Console in RealKeywords and aggregated by URL in SiteCrawler, Analytics Overview, and Custom Reports.
Here is an example from URL Explorer that shows the three sources and different numbers reported from each source:
The nuances of each data source are described below to help you fully understand your visit data.
Log Files
Visit data for actual user activity is collected from your server logs. All organic visits to the site are counted, and there is no concept of sessions, so counts are not filtered or aggregated. All timestamps in your log data are reported in UTC.
Accessing Log Visit Data
Visit data from logs is available in the following locations:
The Visits (Logs) Dataset:
SiteCrawler's Orphan URLs Visits report:
About Bot Visits
Search bots can artificially inflate log visit counts when they attempt to scrape a website for good or bad reasons. While analytics tools usually filter these “fake visits,” they will not be filtered from logs, so you should expect the number of visits reported in log data to be higher than those reported by your analytics source.
Including the client IP addresses in your log data is important to enable Botify to verify the hits are from actual search bots. Please refer to the Botify Logs Integration Process for more information.
Analytics
Analytics data is collected differently than log data in the following ways:
Analytics relies on a piece of code embedded in the page called a tracking pixel, and data collection happens via JavaScript. This means some visits may not be captured if there is an issue with the JavaScript (e.g., it is disabled).
Analytics data uses a period attributed to the same user and session. After a session expires, analytics starts counting clicks as new visits.
The timestamps in your analytics data match the format identified in project settings.
Accessing Analytics Visit Data
Visit data from your analytics provider is integrated with Botify in the following locations:
Analytics Overview Page: Navigate to Analytics > Overview to access your integrated analytics data in the funnel's Convert stage.
Engagement Analytics: Navigate to Analytics > EngagementAnalytics to access reports based on your analytics provider's integrated conversion and revenue analytics data.
Visits Dataset: Two datasets provide visit data when filtering reports or selecting report columns. The “Visits (Analytics)” dataset includes aggregated data on organic visits, and the “Visits” dataset (available in RealKeywords) provides daily data for all channels.
Google Search Console
Google Search Console data is collected from Google's logs containing organic visit search queries and clicks. These are aggregated at the site’s domain or page levels, and this data is imported into Botify through the API and displayed in RealKeywords. Timestamps in GSC data are reported in Pacific Time.
Data Sampling in GSC
GSC samples data for impressions and clicks daily. Navigate to RealKeywords > Keyword Sampling to find your sample rates.
💡 Botify customers on a Pro or Enterprise plan can decrease and, in some cases, eliminate sampling with the Bulk Data Export integration.
Comparison Table
The following summarizes the difference between the three sources of visit data:
Summary and Best Practices
There will always be variations between the three data sources: logs, analytics, and GSC. Keep the following in mind when reporting on visits:
For crawl budget insights: Use visits from logs since you will likely need to analyze the crawl/visit correlation.
For keyword performance: Use visits from GSC with impressions and CTR.
Try to only include canonical/indexable URLs for any analysis that includes visits since GSC aggregates to the canonical.
See also:
Revenue and Conversion Data in Botify