📘 This article describes how to crawl a protected website in Botify.
Overview
You can crawl a website that is not publicly accessible, such as a development or pre-production version of your site. The way to do this depends on the access control method used:
Crawl by a custom HTTP header that only your server knows. See how to add a custom HTTP header to Botify's requests to your server.
Crawling a Password-Protected Site
If your site is password-protected, you can specify a username and password for Botify. Navigate to your project Settings > Advanced Settings tab and enter the login and password in the Access section:
Authentication Type
SiteCrawler uses the username and password provided in project settings for basic access authentication (client-side HTTP basic authentication, as explained here by Wikipedia). Botify adds a line in the HTTP header with these credentials for each page it requests.
Crawling by User Agent
You can crawl a website accessible only to a specific user agent if your website is validated (verified). You can instruct SiteCrawler to use your allowlisted user agent in your project settings. To identify your user agent:
Navigate to Settings > Advanced Settings.
In the Desktop User Agent or Mobile User Agent field, select "Custom", then identify the name of your user agent.
Click Save at the bottom of the page.
Read more about Botify's user agent and customization.
Crawling by IP Address
If your package includes the static IP option, then your account team can enable the option to use a fixed IP in your project settings, and you can allow the IP addresses:
If you do not have the static IP option, you can still get the IP address the crawler is running from for a specific crawl once the crawl has started. It may help if your website will initially respond to any IP address. For instance, you need to allow the crawler's IP address to avoid triggering robot detection systems that will blocklist an IP after detecting a certain volume of activity from that address.
Finding the IP Address of Botify's Crawl
You can find the crawler's IP address by pausing a crawl soon after it begins. This is useful if you do not have the static IP option and must avoid triggering a bot detection system that will blocklist an IP address after detecting intense activity from that address.
The crawl cannot be paused for more than 48 hours, so allowlisting has to be prompt.
To find the IP address:
After starting the crawl, navigate to the Crawl Manager, then click the Watch Live Stats button.
Click the Pause button.
The IP address the crawl is running from is displayed on this live stats page:
Allowlist the crawler's IP address.
Resume the crawl.
👉 We do not provide the entire list of IP addresses Botify can run from because we use a wide IP range, which any technical or security team would find unreasonable to allow in its entirety. You can use the static IP option if you subscribed to it (then Botify will run from one of a few pre-defined IP addresses) or get the IP address the crawl is running from, as described above.