🛠 This document explains the configuration requirements for running SpeedWorkers with Azure Front Door.
How SpeedWorkers Works
SpeedWorkers is a service designed to deliver web pages to search engine crawler bots as fast as possible. It increases your crawl budget (SEO), serving more pages to search engines to index for the same amount of time spent on your website.
SpeedWorkers can prerender your JavaScript pages in advance, at scale, and deliver them to search engines in a few hundred milliseconds. Prerendering pages enables search engines to index your pages faster - in other words, it increases your crawl/ render budget. By storing all pages in its cache, SpeedWorkers delivers long-tail pages as fast as any other page, whereas usual CDNs cannot keep long-tail pages in the cache.
Our service has advanced quality controls to ensure the pages are rendered with all their components.
Configuring Azure Front Door
Configuring Azure Front Door for SpeedWorkers includes the following steps:
Adding a New Origin
To create a new origin for SpeedWorkers in Azure Front Door, create a new Origin Group, as described below.
Log in to your Azure Front Door account.
In the left panel, select Origin groups, and then click the Add button.
In the Origin Group Name field, type SW-Origin-Group, then click the Add an origin link.
Complete the fields as shown below, replacing
BotifyUrl.adn.cloudwith the origin provided by Botify, then click Add.
Verify that the new Origin is listed in the "Origin host name" section.
Deselect the "Enable health probes" option, then click Add.
The new Origin Group for SpeedWorkers is now included in the Origin Group list. This group will not have an associated route; it will be identified as "Unassociated".
Adding a Routing Rule
Adding a routing rule for SpeedWorkers includes the following steps:
Creating a Rule Set
To create a rule set:
Select Rule sets in the left panel, then click the Add button.
Name the rule set "Swrule".
Identify the four conditions shown in the Conditions Table below.
Identify the three actions shown in the Actions Table below.
Confirm the configuration matches what is shown below, then click Save.
Conditions Table
Request header | Header name = User-agent | Operator = Contains | Header value = Botify-bot-sw-test | String transform = 0 selected |
Request method | Operator = Not Equal | Request method = POST |
|
|
Request file extension | Operator = Not Equal | Value = css, js, pnj, jpg, gif, svg, ico, json, xml, woff | String transform = 0 selected |
|
Host name | Operator = Contains | “your main hostname” | String transform = 0 selected |
|
Actions Table
Request header | Operator = Overwrite | Header name = x-sw-adn-token | Header value = Provide by Botify |
|
Request header | Operator = Overwrite | Header name = x-sw-url | Header value = {request_uri} |
|
Route configuration override | Override origin group = Yes | Origin group = SW-Origin-Group | Forwarding protocol = HTTPS only | Caching = Disable |
Associating the Rule Set to a Route
In the list of rule sets, click the ... next to the rule set you just created, then select Associate a Route.
Select an endpoint name.
If other rule sets exist, reorder the list to make the SWRule the first rule to be executed, then click Associate.
The new rule set is now active.
Validating the SpeedWorkers Integration
To validate the integration of SpeedWorkers in your environment, you can send the following requests.
"Always Success" Test
The "always success" test will force SpeedWorkers to return a cache hit even if the page is not in the cache. This test ensures that SpeedWorkers is called and its reply returned to the bot:
Always Success (force a cache hit in SW)
--------------
URL: Your homepage (https://www.mywebsite.com)
Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-success,echo-67674
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify
Expected response:
Status: 200
Body: "Success"
Headers:
X-Ftlcdn-Status: false
X-Sw-Echo: 67674
X-Sw-Passed-Through: true
X-Sw-Status: success
"Cache Miss" Test
The "cache miss" test forces SpeedWorkers to return a cache miss even if it has the page in the cache. This test ensures that when SpeedWorkers can't deliver the page, the request falls back properly:
URL: Your homepage (https://www.mywebsite.com)
Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-notfound,echo-41521
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify
Expected response:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers
"Timeout" Test
The "timeout" test forces SpeedWorkers to delay its response enough to trigger the timeout in your environment. This test ensures that when SpeedWorkers doesn't reply, the request falls back properly:
URL: Your homepage (https://www.mywebsite.com)
Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-timeout,echo-42300
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify
Expected response after several seconds:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers
Revisit Log Files
Please revisit the log files being passed to Botify for ingestion and confirm you are sending us the following:
origin.speedworker.com
cluster.speedworker.com
Troubleshooting
If sending a request to SpeedWorkers doesn’t return the expected response when testing the integration, try the following:
Replace the SpeedWorkers host (origin) with a third-party service like PutsReq, Request Catcher - record HTTPS requests, webhooks, API calls, or Beeceptor - Rest API mocking in seconds. It will help you verify that the call to SW is correct.
Remove the failover rule to get details in the edge diagnostic tools about why the request to SpeedWorkers fails, such as the Akamai reference error.
❗️Before testing with a third-party service, change the website ID and token in the recv snippet to avoid leaking them.