Skip to main content
All CollectionsBotify ActivationSpeedWorkersSpeedWorkers Tech Docs
Deploying SpeedWorkers with Akamai
Deploying SpeedWorkers with Akamai
Updated over 2 weeks ago

🛠 This document explains the configuration requirements for running SpeedWorkers with Akamai.

How SpeedWorkers Works

SpeedWorkers is a service designed to deliver web pages to search engine crawler bots as fast as possible. It increases your crawl budget (SEO), serving more pages to search engines to index for the same amount of time spent on your website.

SpeedWorkers can prerender your JavaScript pages in advance, at scale, and deliver them to search engines in a few hundred milliseconds. Prerendering pages enables search engines to index your pages faster - in other words, it increases your crawl/ render budget. By storing all pages in its cache, SpeedWorkers delivers long-tail pages as fast as any other page, whereas usual CDNs cannot keep long-tail pages in the cache.

Our service has advanced quality controls to ensure the pages are rendered with all their components.

SpeedWorkers Beam Service Installation Concepts

The component serving pages to search engines is called the “SpeedWorkers Beam Service.” In your CDN configuration, you must redirect all eligible search engine bot queries to Botify SpeedWorkers Beam Service (or a subset of those during the test phase, at your discretion).

👉 We may need to discuss with your Akamai support team if your configuration needs additional adjustments we cannot set by ourselves.

Rules

1 - Check the user agent of each incoming query

If the user agent contains the following strings, go to Step 2. Otherwise, call the origin server.

  • googlebot/

  • bingbot/

  • yandexbot/

  • yandexmobilebot/

  • baiduspider/

  • baiduspider+

  • applebot/

  • yeti/

  • botify-bot-sw

👉 Refer to Supported Bots for the full list of user agents.

2 - Call Beam SpeedWorkers API service

Apply the following parameters:

  • URI: Full URL encoded original request (scheme, domain, path, query parameters).

  • website_id: ID of client website provided by Botify.

  • token: Client token provided by Botify.

Ex : https://xxx.speedworkers.com/page?website_id=8DUPBF&token=589sjdjUyUlmkZ2c8mJ0cewgoyvI9oGHfw&uri=https%3A%2F%2Fwww.website.com%2Fproduct%2F149628%2F

3- Receive the response header x-sw-status

  • If x-sw-status is success (= ‘cache hit’): The response from SpeedWorkers can be delivered to bots.

  • If x-sw-status is fail (= ‘cache miss’): Then fall back to your origin server.

Configuring Akamai Property Behaviors‌

Step 1 - Create a failover hostname.

You should already have the main hostname, typically your website domain, targeting an Akamai Edgesuite URL on your property. The purpose is to create a failover hostname, which will be used if Botify cannot serve the page (cache miss or server error). There is no need to associate a DNS entry on your domain because this failover hostname is a dummy name.

Step 1.1 - Create the hostname.

  1. In the “Property Hostnames” section (when editing a version of the property), click Add.
    ​

  2. If your domain is http://www.site.com, then create a hostname www-failover.site.com.
    ​

  3. Click Next.
    ​

  4. Select "IPv4 + IPv6", then click Next.
    ​

  5. Click Submit in the last step. If the Edge Hostname is not generated automatically, create a dummy hostname manually: www-failover.site.com.edgesuite.net or www-failover.site.com.edgekey.net. You should now have two entries in your property hostnames:
    ​


    Since the hostname is a dummy name, there is no need to add a CNAME to the edgesuite on your DNS.

Step 1.2 - Add a property variable.

We need a variable to force Akamai to forward the “if-modified-since” (IMS) header to origins. Add a variable named “IMS”:

Step 1.3 - Change “Forward Host Header” in your default rule.

Now we have a second property on which we may fall back, we need to ensure we always return the same Host header, which is the final hostname.

Change “Forward Host Header” to “Custom Value” and add the name of your main property hostname (e.g., http://www.site.com) to the “Custom Forward Host Header” field.

Before:

After:

Step 2 - Intercept the request.

In your website's Property Manager, navigate to the Property Configuration Settings.

Step 2.1 - Add rule.

Step 2.2 - Insert Conditional Origin rule.

  1. Search for “Conditional Origin Definition” and click on Insert Rule.
    ​

  2. Select “Match All” in the criteria dropdown so that the conditions match different criteria, as shown in the table below.
    ​

Criteria

Fields

User Agent

Operator: is one of
​Example Values*: googlebot/, bingbot/, yandexbot/, yandexmobilebot/, baiduspider/, baiduspider+, applebot/, yeti/, google-inspectiontool/, googleother, botify-bot-sw (for testing and production)

Don’t forget the wildcards! Copy and paste items individually; otherwise, it will create a single match, as shown in the example below.


​* Only include the user agents allowed in your Botify subscription plan. Refer to Supported Bots for the full list of user agents.

Hostname

Operator: “is one of”
​Value: Your main hostname (ex: www.site.com).
​
The second criterion is crucial so the failover origin won’t hit this rule to ensure your main origin will take the request back if SpeedWorkers is down or cannot return the page.

Request Method

Operator: “is not one of”
​Value: “POST”

File Extension

Operator: “is not one of”
​Value: bmp css doc docx eot gif ico ief jpe jpeg jpg js json less m1v map mov mp otf ovg pbm pdf pgm png pnm ppm pps ppt ps qt ras rdf rgb rss svg swf tif tsv ttf txt vcf wav webm woff* xbm xlm xls xml xpdl xpm xwd
​
You can copy the full value line; it will create one item per extension, as shown in the example below.

1090

Individual values with wildcards

1292

File extensions as individual values

Step 2.3 - Set the origin server hostname.

  1. Fill the “Origin Server Hostname” field with the SpeedWorkers Service hostname given by Botify.

  2. Set the following values:

  • Forward Host Header: “Origin Hostname”

  • Supports Gzip Compression: “Yes”

  • Send True Client IP Header: “Yes”

  • True Client IP Header: “X-Sw-Client-IP”

  • Verification Settings: “Third Party Settings”

  • Use SNI TLS Extension: “Yes”

The “Verification Settings” cannot be set to “Use Platform Settings” depending on the SSL configuration. Use your usual settings in this case, but ensure AWS certificates are trusted. If the Verification Settings are set to "Use your own," then the "Use SNI TLS Extension" option must be enabled.

Step 2.4 - Select a Content Provider Code.

Select your Content Provider Code to identify usage, reporting, and billing.

Step 2.5 - Add 'outgoing path' behavior.

  1. Click Add Behavior on the same rule page to change the request path:
    ​

  2. Insert Modify Outgoing Request Path:

    1. Search for “Modify Outgoing Path”.

    2. Click Insert Behavior.
      ​

    3. Select “Replace the entire path” from the Action dropdown list.
      ​

    4. Update “Replace with” with:‌

/page?website_id=YOUR_WEBSITE_ID&token=YOUR_TOKEN&uri=https://{{builtin.AK_HOST}}{{builtin.AK_URL}}

‌Example

/page?website_id=ABCDE&token=hnG76b75fff&uri=https://{{builtin.AK_HOST}}{{builtin.AK_URL}}

Step 2.6 - Add 'timeout' behavior.

  1. Add another behavior:

    1. Click Add Behavior.

    2. Search for “Read Timeout”.
      ​

    3. Set the read timeout to 2 seconds.
      ​

👉 If a timeout failover rule has priority over the SpeedWorkers origin and is not filtered by origin, the failover rule will be called instead of triggering the SpeedWorkers failover rule (redirect to alternate hostname - Step 3). Ensure the SpeedWorkers conditional origin has priority over any generic timeout failover rule.

Step 2.7 - Add Cache Strategy.

We recommend not having your CDN cache pages returned by Speedworkers since they are already in our cache and can be refreshed anytime.

  1. Click Add Behavior, then select “Caching”.
    ​

  2. Select “Bypass Cache” for the caching option.
    ​

Step 2.8 - Don’t cache HTTP Error Responses.

  1. Click Add Behavior, then select “Cache HTTP Error Responses”.

  2. Ensure the Cache HTTP Error Responses option is disabled.
    ​

Step 2.9 - Disable retries on the SW origin requests.

Akamai retries a request three times if it fails due to a network issue or a connection terminated by the Origin by default. If Akamai encounters network issues with SpeedWorkers, it should fall back to the real Origin as soon as possible. The Advanced Rule with the <forward:availability.max-reconnects> tag is necessary to remove the retries

🚧 You must ask your Akamai Project Manager to set this rule, as it can only be set by someone from Akamai. If a retry rule with several retries has priority over this one, then calls to SpeedWorkers origin will be retried! Ensure the SpeedWorkers conditional origin has the priority.

If Tiered Distribution is enabled in the Akamai configuration, then Akamai advises setting the max reconnects to 1 instead of 0.

Step 2.10 - Send the “If-Modified-Since” header to SpeedWorkers.

  1. Click Add Behavior, then select “Set Variable”.

  2. Set the PMUSER_IMS created in Step 1.2 to the value of the “If-Modified-Since” header.
    ​

  3. Click Add Behavior, then select “Modify Incoming Request Header”.

  4. Add a “x-sw-if-modified-since” header with the value of the PMUSER_IMS variable:
    ​

Step 3 - Fallback to the origin server.

If SpeedWorkers cannot serve the page or it is eventually down, we will fall back to your origin server.

Step 3.1 - Add child rule under Conditional original definition.

  1. From the “Add Rule” dropdown, select “Child Rule”. The child rule must be under the “Conditional Origin Definition” rule.
    ​

  2. Search for “Forward the request to an alternate hostname”.
    ​

  3. Confirm the following criteria:‌

    1. The operator is “Match Any”.

    2. The first rule is “Origin Timeout” = “Yes”.

  4. Define a second criterion:

    1. Select “Response Header".

    2. Header name: “x-sw-status”.

    3. Operator: “is not one of”.

    4. Value: “success”.
      ​

  5. Enter the failover property hostname as Alternate Hostname to retry it as a standard request (e.g., www-failover.site.com).
    ​

  6. If the “Downstream Cacheability” behavior was automatically added, ensure it is set to “Don’t allow caching (bust)”.
    ​

Step 4 - Handling Subdomains

If your Akamai property manages subdomains, then some modifications are required. Do not follow these steps if you don’t have subdomains!

❗️If you use SiteShield, only a subset of Parent Akamai servers can communicate with your Origin Server. When a bot performs a request, it hits a Child Akamai server closest to it for performance reasons. That Child server forwards the request to its Parent Server, which then forwards it to the Origin Server. In this environment, using the "Modify Outgoing Request Header" behavior to adjust the Host Header in case of fallback to handle subdomains with the SpeedWorkers integration is not possible. The modified Host header is not forwarded between the Parent and the Child server. They would see a different Host header than the original client request. To fix this issue, your Akamai representative has to set an Advanced Metadata Tag behavior to ensure both ends (Child and Parent) see the same Host header, as shown in the screenshot below.

Step 4.1 - Create one failover hostname per subdomain.

For example, if you have three subdomains:

You must create three failover hostnames:

  • www-failover.example.com

  • blog-failover.example.com

  • store-failover.example.com

The requirement is to ensure you get the original hostname when removing the “-failover” part from the hostname.

Step 4.2 - Create a PMUSER_HOST variable.

Step 4.3 - Add a “Set Variable” behavior in your Default Rule

❗️Do not add this behavior in the SpeedWorkers conditional origin definition rule! Add it in your Default Rule where your origin is called.

  • Variable: PMUSER_HOST

  • Create value from: Expression

  • Expression: {{builtin.AK_HOST}}

  • Operation: Substitute

  • Regex: -failover

  • Replacement:

  • Case Sensitive: On

  • Global Substitution: Off

Like this:

Step 4.4 - Add a “Modify Outgoing Request Header” behavior in your Default Rule

❗️Do not add this behavior to the SpeedWorkers conditional origin definition rule! Add it in your Default Rule where your origin is called.

  • Action: Modify

  • Select Header Name: Other...

  • Custom Header Name: Host

  • New Header Value: {{PMUSER_HOST}}

  • Avoid Duplicate Headers: Yes

Like this:

Ensure this rule is set after the “Set Variable” rule, like this:

Step 4.5 - Add “Forward to Alternate Hostname” child Rules.

  1. In the “Forward to Alternate Hostname” child rule you created in Step 3.1, set the Alternate Hostname to your default origin (e.g., www-failover.example.com).
    ​

  2. Add a “Forward to Alternate Hostname” child Rule to this rule with the following criteria:

    1. If

    2. Hostname

    3. is one of

    4. blog.example.com

  3. Set the Alternate Hostname to: blog-failover.example.com.
    ​

  4. Do the same for the third subdomain with the following criteria:

    1. If

    2. Hostname

    3. is one of

    4. store.example.com

  5. Set the Alternate hostname to: store-failover.example.com.
    ​

Test Configuration

To test the configuration, Botify needs access to Akamai’s staging environment.

Botify uses these IP addresses. You can also do a DNS query on nat-crawler.botify.com to obtain the list, for example:
​dig +short nat-crawler.botify.com.

Validating the SpeedWorkers Integration

To validate the integration of SpeedWorkers in your environment, you can send the following requests.

"Always Success" Test

The "always success" test will force SpeedWorkers to return a cache hit even if the page is not in the cache. This test ensures that SpeedWorkers is called and its reply returned to the bot:

Always Success (force a cache hit in SW)
--------------
URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-success,echo-67674
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: "Success"
Headers:
X-Ftlcdn-Status: false
X-Sw-Echo: 67674
X-Sw-Passed-Through: true
X-Sw-Status: success

"Cache Miss" Test

The "cache miss" test forces SpeedWorkers to return a cache miss even if it has the page in the cache. This test ensures that when SpeedWorkers can't deliver the page, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-notfound,echo-41521
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers

"Timeout" Test

The "timeout" test forces SpeedWorkers to delay its response enough to trigger the timeout in your environment. This test ensures that when SpeedWorkers doesn't reply, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-timeout,echo-42300
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response after several seconds:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers

Revisit Log Files

Please revisit the log files being passed to Botify for ingestion and confirm you are sending us the following:

  • origin.speedworker.com

  • cluster.speedworker.com

Troubleshooting

If sending a request to SpeedWorkers doesn’t return the expected response when testing the integration, try the following:

❗️ Before testing with a third-party service, change the website ID and token in the recv snippet to avoid leaking them.

Did this answer your question?