📘 This article explains how to set up the Generate Sitemaps destination in SmartIndex. SmartIndex is part of Botify's Activation Suite, available with a Botify Pro or Enterprise plan.
Generate Sitemaps is a SmartIndex destination that creates and pushes sitemaps according to your defined criteria.
👉 Generate Sitemaps replaced FastIndex, Botify's previous sitemap generation service. If you currently have FastIndex, it can coexist with Generate Sitemaps for now.
Configuring a Generate Sitemaps destination includes the following steps:
Begin with the instructions in Configuring SmartIndex Destinations.
Identifying the Host Provider and Location
Botify provides two options for hosting sitemaps in SmartIndex:
Option | Description | Configuration Overview |
Hosted by Botify | You delegate the record and host management to Botify. This is the recommended option. | Botify generates CNAME records that you must register with your DNS provider to enable the destination configuration. If you cannot register the CNAME records for any reason, you must use the self-hosted option. |
Self-hosted | You manage the records that we push from SmartIndex into your storage bucket. | Share your AWS S3 or GCP credentials where Botify will push sitemaps. |
❗️ DNS records proxied to Cloudflare are not supported with the Botify-hosted option. You must use “DNS only” records in Cloudflare, or use the self-hosted option in SmartIndex.
Hosted by Botify
We recommend this option since we will deploy and manage the host configuration on your behalf.
To have Botify host your sitemaps:
Provide a name for the sitemap.
Select the "Hosted by Botify" option, then click Configure.
Type a unique host name that is not currently in use by AWS to define where your sitemaps will be hosted, then click Save & generate. This creates CNAME records you must register for the configuration.
Click the copy icon next to the CNAME records to share them with your tech team, then click Save config.
❗️ Registering the CNAME records with your DNS provider is a required step for the Botify-hosted option. If you cannot register the CNAME records, you must use the self-hosted option.
It may take up to 24 hours after registering the CNAME records and saving the SmartIndex configuration before the hosting environment is deployed; a success message will appear when it is complete.
❗️ To modify a domain name while DNS registration is pending, you must delete the configuration and then configure the host with the new name.
Self-Hosted
To host sitemaps on your own:
Provide a name for the sitemap.
Select the "Self-hosted" option, then choose a provider: AWS S3 or Google Cloud Platform.
For AWS S3, identify either your AccessKeyID or RoleARN credentials. For Google Cloud Platform, identify your Service account key.
Click Save & connect.
Enabling Hreflang Management
If your website includes hreflang tags to associate your pages with their localized versions, we recommend enabling the Hreflang Management option to automatically include hreflang annotations in your sitemaps. This helps search engines better understand the relationships between your pages across languages and regions.
To enable hreflang management, slide the toggle to the right:
Identifying Decision Rules
Configure decision rules to determine which pages to push to your sitemaps based on the HTTP status code they return.
Select the sections to push to the destination or push to all sections.
Set the rules for each section. When these conditions are met, pages will be pushed to the destination.
Optionally identify custom decision rules.
Rules
Status Code: The status codes selected here identify the pages to be pushed to the destination based on the latest crawl.
200: 200 status code + no canonical or self-canonical
200 duplicated: 200 status code + non-self-canonical
200 No index: 200 status code + noindex tag
Ignore if not refreshed after (x) days: Pages will not be pushed to sitemaps because they were not crawled in the selected number of days and, therefore, may no longer exist.
Splitting Sitemaps
You can organize your sitemaps by enabling SmartIndex to automatically split them. Use this option instead of creating individual segments in your project segmentation for each sitemap.
💡 SmartIndex uses the page HTML tag, not the URL, to split sitemaps.
You can split sitemaps by the following conditions:
Section: Creates sitemaps for each SmartIndex section in your project.
Language: Creates sitemaps for each language identified by your pages' lang attribute.
Country: Creates sitemaps for each country identified by your pages' lang attribute.
HTML extract: Creates sitemaps for the selected HTML extracts.
👉 You cannot manage HTML extracts within SmartIndex. Visit your project settings to create and manage HTML extracts.
To enable the sitemap split functionality:
Slide the Split sitemaps toggle to the right.
Select up to four conditions to group your sitemaps.
Optionally drag condition blocks to other locations to reorder them. The order impacts how pages are organized into sitemaps, not the order in which they appear in the sitemap.
💡Read some sitemap splitting strategies here.
Setting the Sitemap Generation Rules
The sitemap generation rules identify the conditions when your sitemaps will be created and pushed to the destination.
To set sitemap generation rules:
Select the time when sitemaps will be created.
Identify a change threshold to determine if sitemaps will be pushed to the destination. The threshold is a percentage of change in the number of URLs from the previously pushed sitemap. If the percentage of change exceeds the selected threshold, the newly generated sitemaps will not be pushed to the destination. In the example below, if the number of URLs in the new sitemap increased or decreased by more than 30% from the previous sitemap, the newly generated sitemap will not be pushed to the destination.
Identifying a Scope Filter
You can optionally require URLs to match defined criteria, such as a specific path. To use a scope filter, enable the filter option and identify the criteria to match. When enabled, only URLs matching the criteria will be added to sitemaps.
⚠️ Use caution when identifying a scope filter since SmartIndex does not validate for SEO relevance; therefore, a filter could potentially invalidate custom decision rules.
❗️Remember to save your destination configuration.
See also: