📘 Field descriptions for core crawl data, including HTTP status codes, content properties, crawl timing, indexability, robot directives, and canonical tags.

Overview

Main Crawl fields cover the fundamental properties Botify records for each URL during a crawl: how the server responded, how long it took, whether the page is indexable, and what directives are in place. These fields are the starting point for most crawlability analysis.

Datasets and API collection

Available in: SiteCrawler · RealKeywords
For API users: These fields are available in the Crawl BQL collection.

HTTP Response

Field name	API identifier	Kind	Type	Description
HTTP Status Code	`http_code`	dimension	integer	The HTTP status code returned by the server when Botify requested the page (e.g., 200, 301, 404, 500).
Content-Type	`content_type`	dimension	string	The value in the page Content-Type header (e.g., text/html, application/json).
Content Byte Size	`content_size`	metric	integer	The size of the URL's body as represented in bytes.

Crawl Timing

Field name	API identifier	Kind	Type	Description
HTML Load Time	`delay_last_byte`	metric	integer	The total number of milliseconds it took SiteCrawler to load the page, including Time to First Byte (TTFB).
HTML Download Time	`delay_html_download`	metric	integer	The number of milliseconds it took SiteCrawler to download the HTML content of the page, excluding server response time (TTFB).
Time to First Byte (TTFB)	`delay_first_byte`	metric	integer	The number of milliseconds between SiteCrawler sending a request and receiving the first byte from the server. High TTFB values indicate slow server response times.

Indexability

Field name	API identifier	Kind	Type	Description
Is Compliant	`compliant.is_compliant`	dimension	boolean	Whether the URL meets Botify's indexability criteria. A compliant URL returns a 200 HTTP status code, is not blocked by robots.txt, has no noindex directive, and self-canonicalizes.
Non-Indexable Main Reason	`compliant.main_reason`	dimension	string	The primary reason a URL is not indexable. Common values: http_code_3xx, http_code_4xx, http_code_5xx, noindex, canonical_not_self, robots_blocked.

Robot Directives

Field name	API identifier	Kind	Type	Description
Has Noindex Meta Tag	`metadata.robots.noindex`	dimension	boolean	Whether the page's HTML meta robots tag contains a noindex directive, preventing search engines from indexing it.
Has Nofollow Meta Tag	`metadata.robots.nofollow`	dimension	boolean	Whether the page's HTML meta robots tag contains a nofollow directive, preventing search engines from following its links.
X-Robots-Tag: Noindex	`x_robots_tag.noindex`	dimension	boolean	Whether the page is non-indexable because its HTTP response header contains an X-Robots-Tag with the noindex rule.
X-Robots-Tag: Nofollow	`x_robots_tag.nofollow`	dimension	boolean	Whether bots cannot follow links on this page based on an X-Robots-Tag nofollow rule in its HTTP header.
X-Robots-Tag: All Rules Found Proposed name: X-Robots-Tags: All rules found	`x_robots_tag.value`	dimension	string	All rules found in the page HTTP response header's X-Robots-Tag.

Canonical Tags

Field name	API identifier	Kind	Type	Description
Canonical Points to Self	`canonical.to.equal`	dimension	boolean	Whether the URL's canonical tag points to itself. Pages without a self-canonical are not considered indexable by Botify.
Canonical To	`canonical.to.url`	dimension	string	The target URL specified in the page's canonical tag.
No. of Incoming Canonical Tags	`canonical.from.nb`	metric	integer	The number of other URLs that specify this URL as their canonical target.
Canonicals From	`canonicals.from`	dimension	string	A sample of the URLs that are canonicalizing to this page.

See also:

Main Crawl Fields