Crawling your site with Botify's crawler helps you better understand your website's structure as search engines see it. Running an analysis in Botify is the first step to identifying the key indicators influencing the first step in the SEO conversion funnel - crawl. If Botify can crawl your site well, you will have useful indicators in detailed reports to improve search engine crawls of your website.

Botify crawl reports are available in SiteCrawler after a crawl and its analysis are complete. The evaluation Botify conducts during and after a crawl is called the analysis, and the resulting report is referred to interchangeably as the analysis or crawl report. After the initial crawl in a project, you can compare each analysis to previous analyses to evaluate KPI evolution across the crawls available in your Botify plan.

Crawl settings are in Project Settings, and the Crawl Manager maintains the project's crawl history and provides links to crawl reports and access to live stats during crawls. Access both through the main project navigation bar:

We recommend launching a new crawl every time your website structure or content changes significantly. You should determine your crawl frequency for routine crawls based on your website's update cycles. For content that evolves constantly, consider weekly crawls. Monthly crawls may be sufficient for content that evolves at a slower pace. You can easily manage recurring crawls via projects with automated results comparison over time.

Refer to the following articles to learn more about creating and managing Botify crawls:

See also: