🛠 This document explains the configuration requirements for running SpeedWorkers with Akamai.
How SpeedWorkers Works
SpeedWorkers is a service designed to deliver web pages to search engine crawler bots as fast as possible. It increases your crawl budget (SEO), serving more pages to search engines to index for the same amount of time spent on your website.
SpeedWorkers can prerender your JavaScript pages in advance, at scale, and deliver them to search engines in a few hundred milliseconds. Prerendering pages enables search engines to index your pages faster - in other words, it increases your crawl/ render budget. By storing all pages in its cache, SpeedWorkers delivers long-tail pages as fast as any other page, whereas usual CDNs cannot keep long-tail pages in the cache.
Our service has advanced quality controls to ensure the pages are rendered with all their components.
Setting up Routing for Pre-launch Testing
Before setting the configuration live, you will:
Redirecting Bot Requests to SpeedWorkers
In your CDN configuration, you must redirect all eligible search engine bot queries to Botify SpeedWorkers Beam Service (or a subset of those during the test phase, at your discretion).
👉 We may need to discuss with your Akamai support team if your configuration requires additional adjustments that we cannot make ourselves.
1 - Check the user agent of each incoming query
If the user agent contains the following strings, call the SpeedWorkers Beam API service. Otherwise, call the origin server.
googlebot/
bingbot/
yandexbot/
yandexmobilebot/
baiduspider/
baiduspider+
applebot/
yeti/
botify-bot-sw
👉 Refer to Supported Bots for the full list of user agents.
2 - Call Beam SpeedWorkers API service
Apply the following parameters:
URI: Full URL encoded original request (scheme, domain, path, query parameters).
website_id: ID of client website provided by Botify.
token: Client token provided by Botify.
Ex : https://xxx.speedworkers.com/page?website_id=8DUPBF&token=589sjdjUyUlmkZ2c8mJ0cewgoyvI9oGHfw&uri=https%3A%2F%2Fwww.website.com%2Fproduct%2F149628%2F
3- Receive the response header x-sw-status
If x-sw-status is success (= ‘cache hit’): The response from SpeedWorkers can be delivered to bots.
If x-sw-status is fail (= ‘cache miss’): Then fall back to your origin server.
Routing Requests from Akamai to Botify
Routing requests includes the following steps:
Step 1 - Create a failover hostname
You should already have the main hostname, typically your website domain, targeting an Akamai Edgesuite URL on your property. The purpose is to create a failover hostname, which will be used if Botify cannot serve the page (cache miss or server error). There is no need to associate a DNS entry on your domain because this failover hostname is a dummy name.
Step 1.1 - Create the hostname.
In the “Property Hostnames” section (when editing a version of the property), click Add.
If your domain is http://www.site.com, then create a hostname www-failover.site.com.
Click Next.
Select "IPv4 + IPv6", then click Next.
Click Submit in the last step. If the Edge Hostname is not generated automatically, create a dummy hostname manually: www-failover.site.com.edgesuite.net or www-failover.site.com.edgekey.net. You should now have two entries in your property hostnames:
Since the hostname is a dummy name, there is no need to add a CNAME to the edgesuite on your DNS.
Step 1.2 - Add a property variable.
We need a variable to force Akamai to forward the “if-modified-since” (IMS) header to origins. Add a variable named “IMS”:
Step 1.3 - Change “Forward Host Header” in your default rule.
Now we have a second property on which we may fall back, we need to ensure we always return the same Host header, which is the final hostname.
Change “Forward Host Header” to “Custom Value” and add the name of your main property hostname (e.g., http://www.site.com) to the “Custom Forward Host Header” field.
Before:
After:
Step 2 - Intercept the request.
In your website's Property Manager, navigate to the Property Configuration Settings.
Click the Add Rule button.
Step 2.1 - Insert Conditional Origin rule.
Search for “Conditional Origin Definition” and click on Insert Rule.
Select “Match All” in the criteria dropdown so that the conditions match different criteria, as shown in the table below.
❗️Do not add other user agents in this step!
Criteria | Fields |
User Agent | Operator: is one of |
Hostname | Operator: “is one of” |
Request Method | Operator: “is not one of” |
File Extension | Operator: “is not one of” |
Step 2.2 - Identify the origin server hostname.
Fill the “Origin Server Hostname” field with the SpeedWorkers Service hostname given by Botify.
Set the following values:
Forward Host Header: “Origin Hostname”
Supports Gzip Compression: “Yes”
Send True Client IP Header: “Yes”
True Client IP Header: “X-Sw-Client-IP”
Verification Settings: “Third Party Settings”
Use SNI TLS Extension: “Yes”
Depending on the SSL configuration, the “Verification Settings” cannot be set to “Use Platform Settings.” In this case, use your usual settings. If the Verification Settings are set to "Use your own," then the "Use SNI TLS Extension" option must be enabled.
Step 2.3 - Select a Content Provider Code.
Select your Content Provider Code to identify usage, reporting, and billing.
❗️ Use the same CP code as your default property to ensure SpeedWorkers requests are included in logs delivered through LDS.
Step 2.4 - Add behaviors.
Add an outgoing path and timeout behavior to your Akamai property settings.
On the same rule page, click Add Behavior:
Search for “Modify Outgoing Request Header,” then click Insert Behavior.
From the Action dropdown, select “Replace the entire path”.
Update “Replace with” with:
/page?website_id=YOUR_WEBSITE_ID&token=YOUR_TOKEN&uri=https://{{builtin.AK_HOST}}{{builtin.AK_URL}}
Example
/page?website_id=ABCDE&token=hnG76b75fff&uri=https://{{builtin.AK_HOST}}{{builtin.AK_URL}}
Click the Add Behavior button.
Search for “Read Timeout”, then click Insert Behavior.
Set the read timeout to 2 seconds.
👉 If a timeout failover rule has priority over the SpeedWorkers origin and is not filtered by origin, the failover rule will be called instead of triggering the SpeedWorkers failover rule (redirect to alternate hostname - Step 3). Ensure the SpeedWorkers conditional origin has priority over any generic timeout failover rule.
Step 2.5 - Add the Cache Strategy.
We recommend not having your CDN cache pages returned by Speedworkers since they are already in our cache and can be refreshed anytime.
Click Add Behavior, then select “Caching”.
Select “Bypass Cache” for the caching option.
Step 2.6 - Disable cache for HTTP Error Responses.
Click Add Behavior, then select “Cache HTTP Error Responses”.
Ensure the Cache HTTP Error Responses option is disabled.
Step 2.7 - Disable retries on the SW origin requests.
Akamai retries a request three times if it fails due to a network issue or a connection terminated by the Origin by default. If Akamai encounters network issues with SpeedWorkers, it should fall back to the real Origin as soon as possible. The Advanced Rule with the <forward:availability.max-reconnects> tag is necessary to remove the retries
🚧 You must ask your Akamai Project Manager to set this rule, as it can only be set by someone from Akamai. If a retry rule with several retries has priority over this one, then calls to SpeedWorkers origin will be retried! Ensure the SpeedWorkers conditional origin has the priority.
If Tiered Distribution is enabled in the Akamai configuration, then Akamai advises setting the max reconnects to 1 instead of 0.
Step 2.8 - Send the “If-Modified-Since” header to SpeedWorkers.
Click Add Behavior, then select “Set Variable”.
Set the PMUSER_IMS created in Step 1.2 to the value of the “If-Modified-Since” header.
Click Add Behavior, then select “Modify Incoming Request Header”.
Add a “x-sw-if-modified-since” header with the value of the PMUSER_IMS variable:
Step 3 - Identify Fallback Rules
Set rules in your configuration to enable fallback to your origin server if SpeedWorkers cannot serve the page or is down. If SpeedWorkers cannot serve the page or it is eventually down, we will fall back to your origin server.
Step 3.1 - Add child rule under Conditional original definition.
From the “Add Rule” dropdown, select “Child Rule”. The child rule must be under the “Conditional Origin Definition” rule.
Search for “Forward the request to an alternate hostname”.
Confirm the following criteria:
The operator is “Match Any”.
The first rule is “Origin Timeout” = “Yes”.
Define a second criterion:
Select “Response Header".
Header name: “x-sw-status”.
Operator: “is not one of”.
Value: “success”.
Enter the failover property hostname as Alternate Hostname to retry it as a standard request (e.g., www-failover.site.com).
If the “Downstream Cacheability” behavior was automatically added, ensure it is set to “Don’t allow caching (bust)”.
Step 4 - Handling Subdomains
If your Akamai property manages subdomains, then some modifications are required.
❗️Do not follow these steps if you do not have subdomains!
⚠️ If you use SiteShield, only a subset of Parent Akamai servers can communicate with your Origin Server. When a bot performs a request, it hits a Child Akamai server closest to it for performance reasons. That Child server forwards the request to its Parent Server, which then forwards it to the Origin Server. In this environment, using the "Modify Outgoing Request Header" behavior to adjust the Host Header in case of fallback to handle subdomains with the SpeedWorkers integration is not possible. The modified Host header is not forwarded between the Parent and the Child server. They would see a different Host header than the original client request. To fix this issue, your Akamai representative has to set an Advanced Metadata Tag behavior to ensure both ends (Child and Parent) see the same Host header, as shown in the screenshot below.
Follow these steps to handle subdomains:
Step 4.1 - Create failover hostnames.
Create one failover hostname per subdomain. For example, if you have three subdomains: www.example.com, blog.example.com, store.example.com you must create three failover hostnames:
www-failover.example.com
blog-failover.example.com
store-failover.example.com
The requirement is to ensure you get the original hostname when removing the “-failover” part from the hostname.
Step 4.2 - Create a PMUSER_HOST variable.
Step 4.3 - Add a “Set Variable” behavior in your Default Rule
❗️Do not add this behavior in the SpeedWorkers conditional origin definition rule! Add it in your Default Rule where your origin is called.
Variable: PMUSER_HOST
Create value from: Expression
Expression: {{builtin.AK_HOST}}
Operation: Substitute
Regex: -failover
Replacement:
Case Sensitive: On
Global Substitution: Off
Step 4.4 - Add a “Modify Outgoing Request Header” behavior in your Default Rule
❗️Do not add this behavior to the SpeedWorkers conditional origin definition rule! Add it in your Default Rule where your origin is called.
Action: Modify
Select Header Name: Other...
Custom Header Name: Host
New Header Value: {{PMUSER_HOST}}
Avoid Duplicate Headers: Yes
Ensure this rule is set after the “Set Variable” rule, like this:
Step 4.5 - Add “Forward to Alternate Hostname” child Rules.
In the “Forward to Alternate Hostname” child rule you created in Step 3.1, set the Alternate Hostname to your default origin (e.g., www-failover.example.com).
Add a “Forward to Alternate Hostname” child Rule to this rule with the following criteria:
If
Hostname
is one of
blog.example.com
Set the Alternate Hostname to: blog-failover.example.com.
Do the same for the third subdomain with the following criteria:
If
Hostname
is one of
store.example.com
Set the Alternate hostname to: store-failover.example.com.
Step 5 - Add Configuration for DataStream2
If you are using Datastream 2 to integrate your log data with Botify, configure DataStream in the routing rule you just created, using the same stream as you are using to share logs with Botify.
Testing
Use the following Curl requests to confirm the routing is working properly:
"Always Success" Test
The "always success" test will force SpeedWorkers to return a cache hit even if the page is not in the cache. This test ensures that SpeedWorkers is called and its reply returned to the bot:
Always Success (force a cache hit in SW)
--------------
URL: Your homepage (https://www.mywebsite.com)
Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-success,echo-67674
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify
Expected response:
Status: 200
Body: "Success"
Headers:
X-Ftlcdn-Status: false
X-Sw-Echo: 67674
X-Sw-Passed-Through: true
X-Sw-Status: success
"Cache Miss" Test
The "cache miss" test forces SpeedWorkers to return a cache miss even if it has the page in the cache. This test ensures that when SpeedWorkers can't deliver the page, the request falls back properly:
URL: Your homepage (https://www.mywebsite.com)
Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-notfound,echo-41521
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify
Expected response:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers
"Timeout" Test
The "timeout" test forces SpeedWorkers to delay its response enough to trigger the timeout in your environment. This test ensures that when SpeedWorkers doesn't reply, the request falls back properly:
URL: Your homepage (https://www.mywebsite.com)
Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-timeout,echo-42300
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify
Expected response after several seconds:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers
Revisit Log Files
Please revisit the log files being passed to Botify for ingestion and confirm you are sending us the following:
origin.speedworker.com
cluster.speedworker.com
🛑 Botify will also conduct testing and confirm when successful. Only after receiving confirmation from Botify, proceed to Setting up Routing for Production.
Setting up Routing for Production
To be production-ready, add the remaining user agents for the bots allowed in your Botify subscription plan.
In the Akamai Property Manager, navigate to Property Configuration Settings.
Open the “Conditional Origin Definition” and edit the default rule to add user agents.
Save the configuration.
👉 Include wildcards when identifying user agents. Copy and paste items individually; otherwise, it will create a single match.
Troubleshooting
If sending a request to SpeedWorkers does not return the expected response when testing the integration, try the following:
Replace the SpeedWorkers host (origin) with a third-party service like PutsReq, Request Catcher - record HTTPS requests, webhooks, API calls, or Beeceptor - Rest API mocking in seconds. It will help you verify that the call to SW is correct.
Remove the failover rule to get details in the edge diagnostic tools about why the request to SpeedWorkers fails, such as the Akamai reference error.
❗️ Before testing with a third-party service, change the website ID and token in the recv snippet to avoid leaking them.