📘 This article explains the methods to prove ownership of your website, a requirement for fast crawl speeds and access to advanced settings in Botify.
Overview
Some advanced settings are only available for verified, or validated, websites. When your project domains are validated, you can:
Crawl faster than three pages per second (read more about crawl speed).
Crawl with a custom user-agent.
Override the robots.txt.
The emergency stop feature described in the Crawling Third-Party Sites section is disabled after you validate your website. Read more about why you should validate your site.
If you start a crawl on a site you have not validated, the following warning displays:
Validation Methods
There are several options for validating your website. To select a validation method:
Navigate to Project Settings > Crawler > Domain Validation.
From the Method dropdown list, select one of the following automated validation methods:
Create a specific DNS text field.
Create a file with a specific name at the domain's root (the file can be empty).
Place a specific HTML tag on your website's home page.
Connect your Google account to Botify. If your website is validated in Google Search Console, it will automatically become validated in Botify when you integrate your GSC data with your project.
Click Verify.
Alternatively, click the Contact Us link to send us an email from an email account on your website's domain and include your project's URL in the message. If your email uses a different domain, then include an explanation in your message.
Validating Multiple Domains
You must validate each domain if your site includes multiple domains (www.example1.com / www.example2.com). If you do not validate all allowed domains, the full crawl will be limited to three pages per second. You may choose a different validation method for each domain. You do not need to validate any subdomains (e.g., if you validate example.com, blog.example.com will automatically be validated).
Validating During a Crawl
If you start a crawl before validating your website, the maximum crawl speed is three pages per second, even if you entered a higher speed. Validating your site during a crawl does not automatically increase the crawl speed. After validating your site, you must update speed settings to increase from the default speed.
Crawling Third-Party Sites
If you are crawling a third-party website, you have not validated this site in Botify. When crawling a non-validated website, the Botify crawler automatically adds a link to its user agent to allow the website owner to stop the crawl. The site owner will have access to a link Botify adds to their logs to enable them to stop the crawl or to prevent all future crawls on their domain, which will then be blocklisted in Botify.