🛠 Botify LogAnalyzer requires information from your web server's daily logs and your website's crawl report. This document describes the integration process when Cloudflare hosts the logs.
Overview
There are two methods through which your Cloudflare log data can be integrated with Botify: through Cloudflare's Logpush service and the Cloudflare API. Both approaches require a Cloudflare Enterprise plan.
Configuring Cloudflare Logpush
Cloudflare offers their Enterprise clients a Logpush service that automates log exports to an Amazon S3 bucket.
To configure the Logpush method:
In your Cloudflare account, navigate to Analytics & Logs > Logs.
Click the Add Logpush job button.
On the Select Data Set step, select HTTP requests, then click Next.
Select the following Data Field categories to be included in the HTTP request log, then click Next:
Category | Field |
Client | ClientIP |
ClientRequest | ClientRequestBytes |
ClientRequest | ClientRequestHost |
ClientRequest | ClientRequestMethod |
ClientRequest | ClientRequestPath |
ClientRequest | ClientRequestProtocol |
ClientRequest | ClientRequestReferer |
ClientRequest | ClientRequestScheme |
ClientRequest | ClientRequestSource |
ClientRequest | ClientRequestUri |
ClientRequest | ClientRequestUserAgent |
Edge | EdgeResponseStatus |
Edge | EdgeResponseTimestamp |
5. On the Select Destination step, select Amazon S3, then click Next.
6. Enter your S3 Bucket path and region on the Enter Destination Info step, then click Validate access. If you do not already have an S3 bucket, refer to the AWS instructions to create one. Cloudflare will send a file with an ownership token to the S3 bucket you specified.
7. In the final step, paste the ownership token sent from Cloudflare into the Ownership token field, then click Push.
The Logpush service populates the Logs page.
Cloudflare API
Botify can download your logs by querying the Cloudflare API. To use the Cloudflare API method:
Enable Enterprise Log Sharing (ELS) on Cloudflare. ELS is a service that allows a third-party application to request and fetch logs from Cloudflare over HTTP.
Obtain API credentials:
Cloudflare API Email: Email address associated with your account.
Cloudflare API Key: API key generated on the "My Account" page.
Obtain zone information. The zone corresponds to the scope of logs that Botify will request. Each zone (yourdomain.com) on Cloudflare has a unique number identifier. To locate the Zone ID for a domain, visit the "Reports and Stats" link for the domain and note the ID after the equal sign (e.g. zid=xxxxxx). You can also use the /zones API endpoint to list the Zone IDs.
Send the following to Botify using the appropriate regional Support email address shown below: API Email, API Key, and Zone ID.
Botify will download the logs directly from Cloudflare daily by querying Cloudflare's LogPull API on the scope of the zone you provided.
Contact Support
If you need any assistance, please contact Support.