Skip to main content

Using Cookies and Custom Headers in Botify Crawls

Updated over a year ago

📘 This article explains how to use custom HTTP headers in Botify crawls.

Overview

You can specify custom HTTP headers Botify will use when crawling your website, which is useful in the following scenarios:

  • When you need to simulate the use of cookies.

  • To specify a specific cache-control behavior, such as no-cache.

  • To transmit information your server will use for testing purposes (via a pragma HTTP header).

See the list of request HTTP headers on Wikipedia.

Identifying Custom Headers

To add custom HTTP headers:

  1. Navigate to the Behavior section of Advanced Crawl Settings.

  2. Identify the custom HTTP header field name and value.

  3. Click the + icon to add additional custom HTTP headers.
    ​

    crawl_advancedsettings_custhead.jpg

Custom HTTP headers are added to Botify's standard HTTP headers. None of the crawler's headers are removed; however, you can change some of the crawler's standard HTTP headers in the project settings:

crawl_access_b.jpg

Using Cookies in Headers

Botify's crawler can send a cookie with a fixed value. There are several reasons why cookies may be required to analyze your website. The following are some common scenarios:

Multilingual/Multi-Region Websites

A cookie with the language/region defined for a multilingual/multi-region website is required to access the regional website. Users and robots are redirected to the default version of a multilingual/multi-region website if a cookie is not present.

Session IDs

A cookie maintains a session ID, and when this cookie is not present, the session ID is inserted in the URL. Avoid the following URL results in your analysis by identifying a cookie with a session ID:


​
Session ID in the URL path:

http://www.mywebsite.com/page-description;jsessionid=256JTD2d87J891eVd437q8vn716jh96
​
Session ID in a URL parameter:
​http://www.mywebsite.com/page-description?jsessionid=256JTD2d87J891eVd437q8vn716jh96

Identifying the Cookie Custom Header

To add a cookie to a custom HTTP header:

  1. Follow the instructions above to identify a custom HTTP header.

  2. Type "Cookie" in the Custom HTTP Headers Field Name.

  3. Specify the cookie's name and value in the Custom HTTP Headers field. The field name must be Cookie and the field value [cookie_name]=[cookie_value] (e.g., country=germany). The cookie name ('country' in the example below) is case-sensitive, while the HTTP header name ('Cookie' in HTTP header field name) is not. The custom HTTP header (Cookie: country=germany in this example) is added to each request the crawler sends to your web server.

Multiple Cookies

If you need several cookies, add them on the same line, just as it must appear in the HTTP header. The value of the cookie header can include several cookies (cookie_name=cookie_value) separated by ;. For example, if you need a cookie for the country and a session ID, the custom header value is country=germany; jsessionid=256JTD2d87J891eVd437q8vn716jh96

Botify sends any cookies specified in the project's settings with each request. Botify does not retrieve cookies sent by the web server.

Did this answer your question?