📘 This article explains how Botify compares the pages present across your analyses.
Overview
Botify SiteCrawler reports show changes for all indicators sitewide and at the URL level from one analysis to the next. Also included is the evolution of URLs included in each crawl, those that are new and those that disappeared. Botify may have crawled 100,000 URLs during two compared analyses, but you need additional information to determine if they are the same URLs in both crawls. Find this information in SiteCrawler's Movements report.
Evaluating New and Disappeared URLs
For an overview of URLs that are new since the previous crawl and those that disappeared since the previous crawl, navigate to SiteCrawler > Movements. Read more about the Movements report.
The URLs Distribution chart shows the distribution of pages first discovered in the current crawl (i.e., new URLs), those that existed in the compared crawl but not the current crawl (i.e., disappeared), and those that remain the same across both crawls (i.e., remaining URLs).
This chart links to the "New URLs" and "Disappeared URLs" sections with the same SiteCrawler performance reports filtered for the URLs that appeared for the first time in the current analysis and those that existed in the previous analysis but not the current one.
New URLs
Click on the blue New URLs segment to get details for URLs that appeared on your website between the previous analysis and the current analysis. In the New URLs section, you can investigate whether:
The new URLs include many HTTP errors in the HTTP Codes report.
The new URLs correspond to pages you expect to see on your website. For example, use the URL Explorer on a classifieds website to find out if most new URLs are ads.
Whether new URLs are deep in your website's structure or if your website linking structure and Internal Pagerank distribution changed significantly.
Disappeared URLs
Click on the red Disappeared URLs segment to get details for URLs that appeared in the compared crawl only. In the Disappeared section, you can investigate whether:
Disappeared pages correspond to content that was removed from the website. For example, expired content that is not linked anymore, malformed URLs, etc.
Disappeared pages are gone, or they were not found in the current crawl because they became orphan pages or are much deeper in the website (and, as a result, excluded by the analysis scope).
See also: