LogAnalyzer requires all the log lines from your front-end servers, including cache servers or CDNs, related to crawls by search engine bots and visits from search result pages. Use the following filtering methods to ensure you send the data required to provide the most robust reporting in Botify.

A simple filtering method is to keep all log lines that contain any of the following strings:

To send data for all supported search engines:

Doing this ensures you provide all useful lines to Botify. You will provide additional information, typically lines that contain these strings in fields other than User Agent or Referer. If your log lines use a key-value mechanism (JSON format, Splunk format, etc), you can specifically filter on User-Agent and Referer.

Pattern for detecting bots via their User Agents:

To send user agent data for all our supported search engines:

Pattern for visits:

To send visit data for all our supported search engines:

While Botify does not process any Personally Identifiable Information (PII) to build its analytics or store any PII in its databases, you can remove PII from your log files before sending them to Botify for ingestion. Most CDNs enable you to select or exclude certain fields from the logs and to remove the IP in your CDN's log settings. If you choose to filter logs on your own, please contact Support for assistance.

If you need any assistance, please contact Support using the email address for your region: