Skip to main content

Managing a Crawl

Updated over a year ago

📘 This article explains the options for managing Botify crawls during and after crawl completion.

Overview

Botify provides several options to manage your crawls to meet your needs, including the following:

Access a Crawl in Progress

Crawls in progress are identified in the project tile on the Welcome page, at the top of the project settings page, and in the Crawl Manager. To access a crawl in progress to monitor progress or change settings, choose one of the following:

  • Click the Crawling link on the Welcome page:​
    ​

    crawl_fromwelcome.jpg

  • Navigate to the Crawl Manager from the global project navigation bar:

    crawl_analysisrunning.jpg

Modifying Crawl Settings During a Crawl

While a crawl is in progress, you can modify crawl limits, including the speed, depth, and number of URLs to analyze. Monitor a crawl in progress to evaluate performance by the current settings, and adjust speed settings to meet your needs.

Stopping or Pausing a Crawl

You can stop a crawl at any time, and the analysis will be performed based on the URLs crawled. Your crawl allowance will only be debited for the number of pages crawled when the crawl was stopped. Use this feature with caution since crawls cannot be resumed once stopped. If you are unsure if you want to resume the crawl, use the pause option instead.

To stop or pause a crawl:

  1. Access the crawl in progress.

  2. If you accessed the crawl through the Crawl Manager, click the Watch Live Stats button.
    ​

    crawl_livestats.png

  3. In the Analysis Settings section, click Stop, and then confirm the action; or
    ​

    crawl_stoppause.jpg

  4. Click Pause to pause the crawl immediately. Click Resume when ready to begin the crawl from where it was paused.

If you stop a crawl scheduled on a permanent recurring basis, the next crawl will not occur, and you will need to schedule the permanent recurring crawl again.

Accessing and Managing Crawl History

In addition to crawl settings and analysis statistics for crawls in progress, the Crawl Manager maintains your crawl history according to the limits defined in your Botify plan.

To access a historical crawl report:

  1. Navigate to the Crawl Manager from the global project navigation bar.

  2. Click a report's name in the Analysis column to open the report.
    ​

    crawls_crawlmgr.jpg

  3. The report opens to SiteCrawler's Overview section. Click any section link to navigate to a targeted area of the crawl report, or click Analysis Info to find a summary of the analysis settings.

    crawls_sc_overviewanalysis.jpg

Excluding a Crawl from ActionBoard

To eliminate a crawl analysis from trend data in ActionBoard, slide the toggle in the "Exclude from ActionBoard" column to the right. Revert this anytime by sliding the toggle to the left.

Renaming a Crawl Report

The default report name is the date the crawl began. To rename a historical report:

  1. Click the cog wheel in the report row, then select Add Name.
    ​

    crawls_rename.jpg

  2. Identify a name, then click Save.

    crawls_renamereport.jpg

Deleting a Crawl Report

To permanently remove a crawl report from your crawl history:

  1. Click the cog wheel in the report row, then select Delete.
    ​

    crawls_delete.jpg

    ​

  2. Click Delete Analysis to confirm.​

    crawls_deleteconfirm.jpg

Deleting a report while a crawl or its analysis is in progress will stop the crawl/analysis, the data will be permanently deleted, and your crawl budget will be debited for the number of URLs crawled.

Did this answer your question?