📘 This article describes the reports available in LogAnalzyer, part of Botify's Analytics Suite, available with all Botify plans.
Overview
LogAnalyzer reports provide key metrics that give insight into how search engines are crawling your site and how crawl activity correlates to visitor traffic (when your visit data is included in your log files). LogAnalyzer reports are limited to three-month periods of daily log data since a longer period would include too much data to display.
The following reports are included in LogAnalyzer:
Overview Report
The LogAnalyzer Overview report provides a high-level view of daily crawl and visit data. Click on the KPIs and charts in each section to access the LogAnalyzer URL Explorer with the relevant filters applied.
Overview KPIs
The following KPI totals for the selected period are displayed at the top of the Overview report:
Crawl Volume: The number of times URLs got crawled by the selected bot. Click the View Details link to see the number of times individual URLs were crawled in a URL Explorer report.
Unique URLs: The number of unique URLs crawled by the selected bot. Click the View Details link to see the URLs crawled in a URL Explorer report.
Visits Volume: The number of times URLs received user visits in the selected period. Click the View Details link to see the number of times individual URLs received a visit in a URL Explorer report. Visits Volume includes every redirect hop.
Active URLs: The number of URLs that received at least one user visit in the selected period. Click the View Details link to see the number of times active URLs received a visit in a URL Explorer report.
Overview Visualizations
The following visualizations are included in the Overview report. While the data in these visualizations is specific to the selected bot, as indicated in each chart title, the descriptions below show Google as an example.
URLs Crawled by Google By Day
This chart shows the daily number of Google crawls from your log data with the following metrics:
Crawl Volume: The daily number of crawls.
Unique URLs: The daily number of unique URLs crawled. Since each URL can potentially be crawled several times, this metric provides insight into the volume of unique pages getting crawled by search engines.
New URLs: The number of URLs that received at least one crawl for the first time since Botify started ingesting your log data.
Note: When we first ingest your logs, every URL will be considered new, but this will slowly taper off as we continue ingesting your daily logs.Warning Crawl: If you have identified warning segments to flag specific things you want to be alerted on in crawls, this metric identifies the daily number of unique warning URLs crawled, as identified in the LogAnalyzer Crawl reports. For example, you may have a warning segment to identify if Google is still crawling your previous site when you do a site migration.
Visits From Google By Day
This chart shows the daily number of Google visits from your log data with the following metrics:
Visits Volume: The total number of URLs visited daily.
Active URLs: The daily number of URLs that received at least one visit in the selected period.
New Active URLs: The daily number of URLs that received at least one visit for the first time since Botify started ingesting your log data.
URLs Crawled By Google By Segment By Day
This chart shows the daily crawl volume, unique URLs, or new URLs by bot type for individual segments. Use the dropdowns to select a bot type, metric, and segment.
Visits From Google By Segment By Day
This chart shows the daily visits, active URLs, or new URLs by device type for individual segments. Use the dropdowns to select a device type, metric, and segment.
URLs Crawled By Googlebot User Agent By Day
This chart shows the device Google crawled from each day. Smartphone and Search are the primary bot types (user agents).
Visits From Google By Device By Day
This chart shows the device through which daily user visits originated.
Crawl Reports
LogAnalyzer’s Crawl Reports section includes crawl data by daily distribution, segment distribution, HTTP codes, and resources.
Daily Distribution
The Daily Distribution reports show your daily crawl data by bot type and segment to help you see your crawl budget for a 30-day period and help identify trends. Use the tabs at the top to navigate to the Daily Distribution reports for the following:
Unique URLs Crawled: The total number of individual URLs crawled. Since each URL can potentially be crawled several times, this metric provides insight into the percentage of your site pages getting crawled by search engines.
New Unique URLs Crawled: The number of URLs that received at least one crawl for the first time since Botify started ingesting your log data.
Warning Crawl Volume: If you have identified warning segments to flag specific things you want to be alerted on in crawls, this metric identifies the number of crawls where your warning segment criteria are found.
Warning Unique URLs Crawled: If you have identified warning segments to flag specific things you want to be alerted on in crawls, the total number of URLs crawled where your warning segment criteria are found.
Resources Crawl Volume: The total number of crawls that included resources.
Resources Unique URLs Crawled: The total number of individual URLs crawled that are identified as resources.
Segments Distribution
The Segments Distribution charts show your crawl data for the selected period by segment. You can select up to two segments to include in these charts. The Segments Distribution chart includes the following metrics:
Crawl Volume: The number of crawls in the segment during the selected period.
Unique URLs: The number of unique URLs crawled in the segment during the selected period.
Frequency: The average number of times URLs were crawled over the selected period (number of crawls/number of URLs).
To change the displayed segment in the chart, select another segment from the dropdown list. To display two segments:
Select Advanced selector:
Select a segment.
Optionally, identify the page depth level to include in the report (i.e., the number of clicks away from the home page).
Click Apply.
👉 Selecting a segment in this chart only updates the chart for the segment; this selection is independent of segment selection in the report filters at the top of the page.
HTTP Codes
The HTTP Codes reports can help you understand where you are wasting your crawl budget. Since bots report the status code they get when crawling your site, you can focus on 3xx, 4xx, and 5xx status codes to see where Google is wasting crawl budget. For example, a spike in 3xx codes may prompt an investigation into internal redirects found. In these charts, you can identify the segment with the highest number of 3xx codes and find the specific URLs that returned 3xx codes by clicking into the charts to access the LogAnalyzer URL Explorer report.
Resources
The Resources reports show where Google crawled the resources at separate URLs used by your HTML pages (e.g., images, JavaScript, CSS). Unless it is important for page rendering, you should reduce Google’s crawl of resources to preserve your crawl budget.
💡 Resource crawl data only includes what is identified in your log files. If resources have been filtered out of the log data you provide to Botify, the related crawl data will not be included in these reports.
Visit Reports
The LogAnalyzer Visit reports include visit data from your logs by daily distribution, segment distribution, and HTTP codes. A visit is reported in logs every time a user clicks a link from a search engine results page (SERP), which differs from how Google counts visits with sessions.
Visit data is important because it helps you visualize if there is a correlation between Google crawling and an increase or decrease in visits. Additionally, LogAnalyzer can identify visits to a 301 status code since log files show where a user clicked the page to get redirected. Third-party analytics cannot achieve this since the analytics tag will only fire once it gets to the redirect page, meaning there is no visibility into 301 visits.
Refer to the Crawl report descriptions above since Visit reports are in the same format, differing only in the data type (user visits instead of bot crawls).
Conversion Report
The LogAnalyzer Conversion report shows SEO conversion from crawls to visits by segment. This report provides the best visualization of crawl budget waste since you can see the number of pages crawled within a segment alongside visits. The left side of the chart shows the percentage of crawls, and the right side shows user visits. This helps you easily evaluate if there is potential for optimization when you see Google is spending a lot of time on a segment that gets few visits or users are engaging with segment pages that Google is ignoring.
See also: