Integrating your web traffic data with your Botify project enables analysis of how search engine bots interact with your website. Your web traffic is stored in log files, typically hosted by a CDN, which record every interaction with your website by users and search engines. Integrating this data with Botify’s crawl data helps you realize the full benefit of Botify, providing key metrics that offer insight into how search engines crawl your site and how crawl activity correlates with visitor traffic.

The integration process includes the following steps:

After the first delivery and confirmation, data is automatically updated in your Botify project daily.

We need all your CDN log files. If you do not use a CDN or part of your traffic goes directly to your website, we need the logs from all your front-end web servers that receive direct requests from crawlers and users.

For each log line, include the following fields:

For non-IIS servers, we recommend using the Apache Combined Log Format without any change if you can configure it on your servers:

Include the following types of data in your log files, filtering content where needed:

In the logs, we detect that a bot explored a page through the user agent. We activate the detection of Googlebot, Google subbots, and AI bots by default. If you are on a Botify Pro plan, we also activate Bing. Contact us if you need to activate Yandex, Naver, or Baidu bot analysis.

The following is a log line example for a crawl from Googlebot:

Botify detects a visit from a human user through the Referer found in the logs. While not required, using the visits feature in Botify is strongly recommended to directly compare visits according to logs with crawls from search engines. To measure organic visits using your log files, you must include the Referer field in your server log data.

Here is a log line example for an organic visit (where Google.fr referred the user):

Processing your web server log files allows us to analyze the main search engine bots' exploration of your website and count SEO visits sent from search engine result pages daily.

Within the server log files, Botify uses only a subset of log lines related to the search engines you want to monitor. You may provide us with this data subset or the full log file content.

There is no theoretical limit to the number of logs you may upload. If you expect to upload more than 200GB of logs per day, let the Botify Support team know in advance. Our system handles files up to 8GB each. If you deliver larger files, we will need to add a split step before the processing step, which will add delays for the initial setup and before every daily update.

At the start of a log file processing, Botify runs a filtering process that discards all lines that are not crawls or visits and anonymizes all IP addresses for visit lines. The result is an anonymized file reduced to its minimal size that is processed again to obtain the SEO data displayed in the application. You can choose whether this filtering and anonymization process occurs in the European Union or the United States.

You may filter and anonymize logs yourself before sending them to us to reduce the volume of sent logs and anonymize them on your side by doing the following:

Botify offers two methods for delivering log files:

Example:

If you deliver logs daily, you must provide the previous day's logs each night. We take your new log files into account every six hours, but you can upload new log files at any time interval. The delay in making your data available in the logs report depends on your data volume, but it is usually less than one hour after your logs were taken into account if segmentation has not changed since the previous processing.

As soon as the first files arrive, Botify validates the log files to ensure the following:

After the first log report is delivered to your Botify project, we ask you to validate the following:

Confirm the volume of analyzed data corresponds to the expected volume of data. Please work with your Botify account team to inspect the following and confirm the volumes correspond to your expectations:

Verify the URLs and domains included in your LogAnalyzer dashboard match your expectations. In the URL Explorer, filter on “URL does not contain” your expected domains to ensure all URLs match your scope. The Botify Professional Services team can assist in this validation and the “Logs QA” process.

We need the logs from your CDN for all web traffic that goes through your CDN. Please refer to the appropriate guide for additional requirements specific to each provider:

If you need any assistance, please contact Support.

See also: