📘 This article describes how to crawl a protected website in Botify.
Overview
You can crawl a website that is not publicly accessible, such as a development or pre-production version of your site. The way to do this depends on the access control method used:
Crawl by a custom HTTP header that only your server knows. See how to add a custom HTTP header to Botify's requests to your server.
Crawling a Password-Protected Site
If your site is password-protected, you can specify a username and password for Botify. Navigate to your project Settings > Advanced Settings tab and enter the login and password in the Access section:
Authentication Type
SiteCrawler uses the username and password provided in project settings for basic access authentication (client-side HTTP basic authentication, as explained here by Wikipedia). Botify adds a line in the HTTP header with these credentials for each page it requests.
Crawling by User Agent
You can crawl a website accessible only to a specific user agent if your website is validated (verified). You can instruct SiteCrawler to use your allowlisted user agent in your project settings. To identify your user agent:
Navigate to Settings > Advanced Settings.
In the Desktop User Agent or Mobile User Agent field, select "Custom", then identify the name of your user agent.
Click Save at the bottom of the page.
Read more about Botify's user agent and customization.
See also: