📘 This article explains how to schedule Botify crawls.

Overview

You can schedule Botify crawls to automatically occur at your preferred time on a one-time or recurring basis. Access crawl scheduling options by navigating to Project Settings > Crawler > Scheduling tab:

Scheduling One-Time Crawls

You can schedule a single crawl or several crawls to begin at different dates and times.

To schedule a single crawl:

Enter the day and time to start the crawl in the Start Date fields.
Select None for the Repeat Frequency.
Click Save. The scheduled date and time are displayed with the crawl status. An email notification will be sent 24 hours before the crawl's scheduled date, and another notification sent when the crawl starts.

Cancelling a Scheduled Crawl

You can cancel a scheduled crawl any time before the crawl begins.

To cancel a scheduled crawl:

On the Scheduling page, click the Delete button next to the scheduled crawl.
Click Yes, Delete Scheduled Crawl to confirm.

Scheduling Repeat Crawls

There are two types of recurring crawls you can schedule:

Scheduling Crawls at Repeat Intervals

You can schedule crawls to occur daily, weekly, or monthly.

To schedule a repeat crawl:

Select the Repeat option in the Repeat Frequency section.
Identify the number to represent the frequency and select the corresponding time units to define the schedule (e.g., every 1 week on Sunday at 11:00).
Click Save. For weekly and monthly crawls, an email notification will be sent 24 hours before the crawl's scheduled date, and another notification will be sent when the crawl starts.

Scheduling Permanent Recurring Crawls

The Permanent Recurring crawl option is a method to run crawls in succession continuously; a new crawl starts as soon as the previous analysis is complete and the report is delivered. Permanent Recurring crawls are useful in the following scenarios:

For websites with a high content rotation rate, such as classifieds, e-commerce websites with a fast-changing product catalog, or a blog with many daily publications.
When the website is undergoing significant changes that will take time to be reflected on the website, and you need visibility into these changes.
For large websites, which take several days or weeks to crawl, you want to ensure one crawl is finished before starting the next.

To schedule permanent recurring crawls:

Select the Permanent Recurring option in the Repeat Frequency section.
In the Delay text field, define the number of minutes to wait between the completion of a report and the start of the new crawl. The minimum delay is 60 minutes.
Click Save.
Schedule a one-time crawl to identify when the permanent recurring crawls will begin.

If a crawl is already running when a crawl is scheduled to start, the scheduled crawl will not start and you will be notified by email. The next analysis in the schedule will start normally. This can occur if the previous crawl is not finished (e.g., a daily scheduled crawl takes more than 24 hours to analyze your website with the current speed settings), or if another analysis was started manually.

Disabling Repeat Crawls

You can disable repeat crawls anytime.

For repeat crawls:

Select None for Recurring Type.
Click Save.

To disable permanent recurring crawls, stop the crawl in progress. This resets the permanent recurring crawl configuration, and you must schedule a new permanent recurring crawl to start it again.

👉 Stopping a crawl configured with a repeat crawl will not affect the next scheduled crawl.