📘 This article explains how to manually start a crawl of your website with Botify's crawler.
Overview
You can manually start a Botify crawl from a new or existing project or schedule recurring crawls.
To ensure a fast crawl speed, consider validating your website before starting a crawl if you have not already done this.
Starting a Crawl Manually
To manually start a crawl:
If you have never initiated a crawl, click the project name on your Welcome page:
If your project has previous crawls, click the cog wheel icon in the global project navigation bar, then select Crawl Manager.
Optionally, click the View Crawl Settings link to see a summary of the current settings, or click the Settings link next to the button to edit the project settings.
Click the Yes! Start Now button on the project settings page.
The crawl begins, and a summary of the crawl settings is displayed with the estimated crawl completion time.
While you can use the Botify crawler to crawl a site you do not own, the site owner can stop the crawl at any time.
Starting an Ad-Hoc Crawl
To launch a limited, short-term crawl for testing purposes, create an ad-hoc project and then follow the steps above to start the crawl.
Monitoring Crawl Status
Your analysis settings and crawl statistics are displayed during the crawl, including the following real-time details over the last two hours:
Crawl speed (number of pages per second).
Bandwidth utilization on HTTP 200 pages (i.e., those that return content).
Average response time: If your crawl is executing JavaScript, the average time to render each page is displayed.
The total number of pages crawled and the total discovered (crawled + in the queue).
New URLs discovered minute-by-minute.
HTTP status codes returned (overall and minute-by-minute).
To access the real-time statistics over the last two hours, navigate to Project Settings > Crawl Manager, then click Watch Live Stats:
Alternatively, click the Crawling link from the Welcome page:
All your advanced indicators will be available after the crawl and analysis completion in the full crawl report in SiteCrawler. The historical reports available according to your Botify plan are accessible in the Crawl Manager.
The time required for a crawl and analysis depends on the number of pages and server performance. Most analyses take a few hours, but they can vary from minutes for a few thousand pages to several days for millions of pages. When the crawl and analysis are complete, you will receive an email with a link to the SiteCrawler report.
See also: