Skip to main content

Deploying SpeedWorkers with Akamai - Recommended Implementation

Updated over 5 months ago

🛠 This document explains the configuration requirements for running SpeedWorkers with Akamai in our recommended implementation.

How SpeedWorkers Works

SpeedWorkers is a service designed to deliver web pages to search engine crawler bots as fast as possible. It increases your crawl budget (SEO), serving more pages to search engines to index for the same amount of time spent on your website.

SpeedWorkers can prerender your JavaScript pages in advance, at scale, and deliver them to search engines in a few hundred milliseconds. Prerendering pages enables search engines to index your pages faster - in other words, it increases your crawl/render budget. By storing all pages in its cache, SpeedWorkers delivers long-tail pages as fast as any other page, whereas usual CDNs cannot keep long-tail pages in the cache.

Speedworkers Architecture with Akamai

  • Botify will serve pages to search engines instead of your regular servers.

  • The CDN will route all requests from a search engine to Botify’s origin instead of your regular server's origin.

  • Botify's origin (Cloudflare worker) will check if SpeedWorkers has the requested page in its cache.
    - If yes, it will deliver the page from the Botify cache.
    - If not, the request to your CDN will fall back with a specific user-agent fallback (Mozilla/5.0 compatible; botify; http://botify.com) to avoid a loop.

Our service has advanced quality controls to ensure the pages are rendered with all their components.

Configuring Akamai Property Behaviors‌

The following describes configuring your environment for the Recommended SpeedWorkers implementation, where SpeedWorkers will retrieve pages not in its cache from your origin server. To configure your environment to handle this fallback on your own, please refer to the Advanced Implementation guide.

Summary of the following configuration:

  • Route bot traffic to the Botify origin based on a user-agent filter (Step 1).

  • Add x-sw-url request header, which contains the value of the URL requested.

  • Add x-sw-adn-token request header, which contains the authentication token provided by Botify.

  • Do not use the CDN cache for those requests.

Configuring the Akamai property behaviors includes the following steps:

  1. Confirming the hostname

  2. Intercepting the request

  3. Identifying the origin server settings

  4. Adding behaviors

  5. Disabling retries on the SpeedWorkers origin requests

1: Confirm the Hostname

You should already have the main hostname on your property, typically your website domain, targeting an Akamai Edgesuite URL.

2: Intercept the Request

  1. In the Akamai Property Manager, navigate to Property Configuration Settings.

  2. Click the Add Rule button.
    ​

  3. Search for “Conditional Origin Definition”, then click Insert Rule.
    ​

  4. Select “Match All” in the criteria dropdown so that the conditions match different criteria, as shown in the table below:
    ​

Criteria

Fields

User Agent

Operator: is one of
​Example Values*: googlebot/, bingbot/, botify-bot-sw (for testing and production)

* Only include the user agents allowed in your Botify subscription plan. Refer to Supported Bots for the full list of user agents.

Hostname

Operator: “is one of”
​Value: Your main hostname (ex: www.site.com).
​
The second criterion is crucial so the failover origin won’t hit this rule to ensure your main origin will take the request back if SpeedWorkers is down or cannot return the page.

Request Method

Operator: “is not one of”
​Value: “POST”

File Extension

Operator: “is not one of”
​Value: bmp css doc docx eot gif ico ief jpe jpeg jpg js json less m1v map mov mp otf ovg pbm pdf pgm png pnm ppm pps ppt ps qt ras rdf rgb rss svg swf tif tsv ttf txt vcf wav webm woff* xbm xlm xls xml xpdl xpm xwd
​
You can copy the full value line; it will create one item per extension, as shown in the example below.

👉 Remember the wildcards when identifying user agents. Copy and paste items individually; otherwise, it will create a single match, as shown in the example below:

1090

Individual values with wildcards

1292

File extensions as individual values

3: Identify the Origin Server Settings

  1. Fill the "Origin Server Hostname" field with the SpeedWorkers Service hostname given by Botify.

  2. Confirm the following settings:

    • Forward Host Header: “Origin Hostname”

    • Supports Gzip Compression: “Yes”

    • Send True Client IP Header: “Yes”

    • True Client IP Header: “X-Sw-Client-IP”

    • Verification Settings: "Choose Your Own"

    • Use SNI TLS extension: “Yes”
      ​


      Depending on the SSL configuration, the “Verification Settings” cannot be set to “Use Platform Settings.” In this case, use your usual settings. If the Verification Settings are set to "Use your own," then the "Use SNI TLS Extension" option must be enabled.
      ​

  3. Select your Content Provider Code to identify usage, reporting, and billing.

4: Add Behaviors

Add the following behaviors to your Akamai property settings:

Modify Outgoing Request Headers

  1. On the same rule page, click Add Behavior.
    ​

  2. Search for “Modify Outgoing Request Header,” then click Insert Behavior.
    ​

  3. From the Action dropdown, select "Modify".

  4. From the Select Header Name dropdown, select "Other...", then select Yes for the "Avoid Duplicate Headers" option.

  5. In the Custom Header Name field, type x-sw-url.
    ​

  6. In the Header Value field, type the following:
    ​https://{{builtin.AK_HOST}}{{builtin.AK_URL}}

  7. On the same rule page, click Add Behavior.

  8. Search for “Modify Outgoing Request Header”, and then click Insert Behavior.

  9. From the Action dropdown, select "Modify".

  10. From the Select Header Name dropdown, select “Other...”, and then select Yes for the "Avoid Duplicate Headers" option.

  11. In the Custom Header Name field, type x-sw-adn-token.

  12. In the Header Value field, type or paste your token.
    ​


    ​

Add a Timeout Behavior

  1. Click the Add Behavior button.

  2. Search for “Read Timeout”, then click Insert Behavior.
    ​

  3. Set the read timeout to 15 seconds.
    ​

Add the Cache Strategy

We recommend not having your CDN cache pages returned by Speedworkers since they are already in our cache and can be refreshed anytime.

  1. Click the Add Behavior button, then search for “Caching”.

  2. Click Insert Behavior.
    ​

  3. In the Caching option field, select “Bypass Cache”.
    ​

Disable Caching for HTTP Error Responses

  1. Click Add Behavior, then select “Cache HTTP Error Responses”.

  2. Confirm this option is disabled.

5: Disable Retries on the SpeedWorkers Origin Requests (optional)

Akamai retries a request three times if it fails due to a network issue or a connection terminated by the Origin by default. If Akamai encounters network issues with SpeedWorkers, it should fall back to the real Origin as soon as possible. The Advanced Rule with the <forward:availability.max-reconnects> tag is necessary to remove the retries

🚧 You must ask your Akamai Project Manager to set this rule, as it can only be set by someone from Akamai. If a retry rule with several retries has priority over this one, then calls to SpeedWorkers origin will be retried! Ensure the SpeedWorkers conditional origin has the priority.

If Tiered Distribution is enabled in the Akamai configuration, then Akamai advises setting the max reconnects to 1 instead of 0.

IPs Whitelisting

To allow crawling requests to cache your website without any issue, we recommend you whitelist Botify's user agents and/or IP addresses.

UAs (can be changed): We recommend adding a security header to secure requests. We can create and share it with you.

Cloudflare Worker IPs and UA: To allow fallback requests to access your website in case a page is not in Speedworkers cache, we recommend you whitelist our UA and/or IPs.


​UA (can be changed): Mozilla/5.0 (compatible; botify; http://botify.com) fallback
We recommend adding a security header to secure requests. We can create and share it with you.

Validating the SpeedWorkers Integration

To validate the integration of SpeedWorkers in your environment, you can send the following requests.

"Always Success" Test

The "always success" test will force SpeedWorkers to return a cache hit even if the page is not in the cache. This test ensures that SpeedWorkers is called and its reply returned to the bot:

Always Success (force a cache hit in SW)
--------------
URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-success,echo-67674
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: "Success"
Headers:
X-Ftlcdn-Status: false
X-Sw-Echo: 67674
X-Sw-Passed-Through: true
X-Sw-Status: success

"Cache Miss" Test

The "cache miss" test forces SpeedWorkers to return a cache miss even if it has the page in the cache. This test ensures that when SpeedWorkers can't deliver the page, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-notfound,echo-41521
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: your homepage
Headers:
X-Sw-Adn: success

"Timeout" Test

The "timeout" test forces SpeedWorkers to delay its response enough to trigger the timeout in your environment. This test ensures that when SpeedWorkers doesn't reply, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-timeout,echo-42300
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response after several seconds:
Status: 200
Body: your homepage
Headers:
X-Sw-Adn: success


​
​

Did this answer your question?