These are common questions that may help with basic troubleshooting across Botify. Also, refer to the links at the bottom of this page for FAQs within specific capabilities.

Setting up a recurring crawl does not start the first analysis of the series. You must define when to start the first analysis by setting the repeat frequency.

Please contact Support to resend an invite to a project.

Hreflang tag analysis is triggered when the tags are discovered on your site during Botify's crawl; no setup is required. Results are shown in the Internationalization section of SiteCrawler's Distribution report.

No. Google Analytics filters out bot activity and Botify does not load GA tags when crawling, so GA is unaware that Botify has been to the site, even when rendering JavaScript.

Botify cannot determine an exact end time because factors are variable, but you can find the estimated remaining crawl time by navigating to Crawl Manager > Watch Live Stats:

There is nothing you need to do in this situation since the Botify Support team is automatically notified of a problem when this occurs, and they will resume your analysis. The unexpected can sometimes happen, depending on website specifics.

If you validate your website after a crawl starts, the maximum speed is three pages per second, even if you entered a higher speed. The higher crawl speed will not be applied until you update and save the speed settings.

URLs crawled by Google are crawls by Googlebot, and visits are organic visits from Google search engine result pages.

These numbers would ideally match, indicating that Botify crawled all pages defined in the scope of your project settings. When they do not match, one of the following occurred:

This is expected since non-branded keywords exclude anonymized queries.

This is to align the days of the week. For example, your report may show 16 January 2023 - 15 January 2024. Many sites experience the bulk of their traffic on either weekdays or weekends, so there may be a discrepancy if comparing a Monday-Friday to a Tuesday-Saturday.

There are many reasons why this may occur. The most common cause is a redirect where the crawl uses a start URL within the allowed domains in the project settings but redirects to another disallowed domain or subdomain. It could also redirect to a disallowed protocol: for example, if the start URL http://www.mywebsite.com redirects to https://www.mywebsite.com and the HTTPS protocol is disallowed.

You can determine if a redirect was the cause by checking if the HTTP status code of the single URL crawled was HTTP 301 (permanent redirect) or HTTP 302 (temporary redirect). Check the redirect target and the allowed domains to confirm.

The redirect could also be a meta-redirect implemented in JavaScript (not executed by the crawler), which means the only URL crawled will return HTTP 200 (OK), but the page will contain no link for the crawler to follow.

Other potential reasons include:

Search engines evaluate pages using the full URL, including the query string (the portion after the ?). As a result, "http://www.domain.com/page1.html?xxx and "http://www.domain.com/page1.html?yyy" are two distinct pages, which may have similar content or not, depending on what the query string does and how it affects the page content. URLs with audience tracking parameters will not change the page content and will be duplicates of the URL without parameters. Parameters that change the page content can either generate partial duplicates (e.g., the number of items displayed in a list) or no duplicates at all (e.g., pagination parameter). Even if URLs are consolidated appropriately with a canonical tag, search engines, and Botify will still consider them separate URLs.

Botify counts all characters present in the Title tag in the code, including spaces and special characters. You can check for extra spaces in a tag by adding one of the {tag} length fields (e.g., Title Length) as a report column in the URL Explorer and viewing the HTML source:

You must enable Sitemaps analysis in crawl settings to access the Sitemaps report.

No. SiteCrawler does not crawl links containing # in the URL. You can, however, find jump links that generate traffic by creating a simple RealKeywords report. Use the filter "Full URL contains #" and add columns for impressions, clicks, CTR, and average position.
​

Although this rarely happens, timeouts can occur, particularly when there is a lot of data. The failure will be shown in the Botify DataExports if this occurs.

While the export size limit is 1,000,000 URLs, the number of lines in the exported file can be far greater if you export a table with a multi-valued column (i.e., a column with a list of data for each URL from the left column). In that case, the export will include several lines per URL. For example, if you are displaying a sample of Inlinks (source pages, where the URLs are linked from), the sample can include up to 300 inlinks per URL, which is up to 3M exported lines. The export will be of the following form:

If your URL Explorer result is too big for a single export, the best way to solve the problem is to do several smaller exports using additional filters. For instance, if you were exporting all URLs with duplicate H1 tags, here are possible approaches:

Start typing what you are looking for in the search field and the list to choose from will be shortened to a selection that contains what you typed. For instance, type "http" for HTTP code, "indexable" for fields related to indexable or non-indexable URLs, "H1" for fields related to H1 tags, "anchor" for fields related to links anchor texts, "inlinks" or "outlinks" for any field related to incoming/outgoing links.

There are two ways to remove a report column:

See also: