Skip to main content

Deploying SpeedWorkers with Akamai - Recommended Implementation

Updated today

🛠 This document explains the configuration requirements for running SpeedWorkers with Akamai in our recommended implementation.

How SpeedWorkers Works

SpeedWorkers is a service designed to deliver web pages to search engine crawler bots as fast as possible. It increases your crawl budget (SEO), serving more pages to search engines to index for the same amount of time spent on your website.

SpeedWorkers can prerender your JavaScript pages in advance, at scale, and deliver them to search engines in a few hundred milliseconds. Prerendering pages enables search engines to index your pages faster - in other words, it increases your crawl/render budget. By storing all pages in its cache, SpeedWorkers delivers long-tail pages as fast as any other page, whereas usual CDNs cannot keep long-tail pages in the cache.

Speedworkers Architecture with Akamai

  • Botify will serve pages to search engines instead of your regular servers.

  • The CDN will route all requests from a search engine to Botify’s origin instead of your regular server's origin.

  • Botify's origin (Cloudflare worker) will check if SpeedWorkers has the requested page in its cache.
    - If yes, it will deliver the page from the Botify cache.
    - If not, the request to your CDN will fall back with a specific user-agent fallback (Mozilla/5.0 compatible; botify; http://botify.com) to avoid a loop.

Our service has advanced quality controls to ensure the pages are rendered with all their components.

About this Configuration

The following describes configuring your environment for the Recommended SpeedWorkers implementation, where SpeedWorkers will retrieve pages not in its cache from your origin server. To configure your environment to handle this fallback on your own, please refer to the Advanced Implementation guide.

The following summarizes this configuration:

  • Route bot traffic to the Botify origin based on a user-agent filter (Step 1).

  • Add x-sw-url request header, which contains the value of the URL requested.

  • Add x-sw-adn-token request header, which contains the authentication token provided by Botify.

  • Do not use the CDN cache for those requests.

Setting up Routing for Pre-Launch Testing

Before setting the configuration live, you will:

Configuring the Akamai Property Behaviors

Configuring the Akamai property behaviors includes the following steps:

  1. Confirming the hostname

  2. Intercepting the request

  3. Identifying the origin server settings

  4. Adding behaviors

  5. Disabling retries on the SpeedWorkers origin requests

1: Confirm the Hostname

You should already have the main hostname on your property, typically your website domain, targeting an Akamai Edgesuite URL.

2: Intercept the Request

  1. In the Akamai Property Manager, navigate to Property Configuration Settings.

  2. Click the Add Rule button.
    ​

  3. Search for “Conditional Origin Definition”, then click Insert Rule.
    ​

  4. Select “Match All” in the criteria dropdown so that the conditions match different criteria, as shown in the table below:
    ​

Criteria

Fields

User Agent

Operator: is one of
​Example Values*: botify-bot-sw (for testing)

Hostname

Operator: “is one of”
​Value: Your main hostname (ex: www.site.com).
​
The second criterion is crucial so the failover origin won’t hit this rule to ensure your main origin will take the request back if SpeedWorkers is down or cannot return the page.

Request Method

Operator: “is not one of”
​Value: “POST”

File Extension

Operator: “is not one of”
​Value: bmp css doc docx eot gif ico ief jpe jpeg jpg js json less m1v map mov mp otf ovg pbm pdf pgm png pnm ppm pps ppt ps qt ras rdf rgb rss svg swf tif tsv ttf txt vcf wav webm woff* xbm xlm xls xml xpdl xpm xwd
​
💡 You can copy the full value line; it will create one item per extension, as shown in the example below.

1292

File extensions as individual values

3: Identify the Origin Server Settings

  1. Fill the "Origin Server Hostname" field with the SpeedWorkers Service hostname given by Botify.

  2. Confirm the following settings:

    • Forward Host Header: “Origin Hostname”

    • Supports Gzip Compression: “Yes”

    • Send True Client IP Header: “Yes”

    • True Client IP Header: “X-Sw-Client-IP”

    • Verification Settings: "Choose Your Own"

    • Use SNI TLS extension: “Yes”
      ​


      Depending on the SSL configuration, the “Verification Settings” cannot be set to “Use Platform Settings.” In this case, use your usual settings. If the Verification Settings are set to "Use your own," then the "Use SNI TLS Extension" option must be enabled.
      ​

  3. Select your Content Provider Code to identify usage, reporting, and billing.

❗️ Use the same CP code as your default property to ensure SpeedWorkers requests are included in logs delivered through LDS.

4: Add Behaviors

Add the following behaviors to your Akamai property settings:

Modify Outgoing Request Headers

  1. On the same rule page, click Add Behavior.
    ​

  2. Search for “Modify Outgoing Request Header,” then click Insert Behavior.
    ​

  3. From the Action dropdown, select "Modify".

  4. From the Select Header Name dropdown, select "Other...", then select Yes for the "Avoid Duplicate Headers" option.

  5. In the Custom Header Name field, type x-sw-url.
    ​

  6. In the Header Value field, type the following:
    ​https://{{builtin.AK_HOST}}{{builtin.AK_URL}}

  7. On the same rule page, click Add Behavior.

  8. Search for “Modify Outgoing Request Header”, and then click Insert Behavior.

  9. From the Action dropdown, select "Modify".

  10. From the Select Header Name dropdown, select “Other...”, and then select Yes for the "Avoid Duplicate Headers" option.

  11. In the Custom Header Name field, type x-sw-adn-token.

  12. In the Header Value field, type or paste your token.
    ​

Add a Timeout Behavior

  1. Click the Add Behavior button.

  2. Search for “Read Timeout”, then click Insert Behavior.
    ​

  3. Set the read timeout to 15 seconds.
    ​

Add the Cache Strategy

We recommend not having your CDN cache pages returned by Speedworkers since they are already in our cache and can be refreshed anytime.

  1. Click the Add Behavior button, then search for “Caching”.

  2. Click Insert Behavior.
    ​

  3. In the Caching option field, select “Bypass Cache”.
    ​

Disable Caching for HTTP Error Responses

  1. Click Add Behavior, then select “Cache HTTP Error Responses”.

  2. Confirm this option is disabled.

5: Disable Retries on the SpeedWorkers Origin Requests (optional)

Akamai retries a request three times if it fails due to a network issue or a connection terminated by the Origin by default. If Akamai encounters network issues with SpeedWorkers, it should fall back to the real Origin as soon as possible. The Advanced Rule with the <forward:availability.max-reconnects> tag is necessary to remove the retries

🚧 You must ask your Akamai Project Manager to set this rule, as it can only be set by someone from Akamai. If a retry rule with several retries has priority over this one, then calls to SpeedWorkers origin will be retried! Ensure the SpeedWorkers conditional origin has the priority.

If Tiered Distribution is enabled in the Akamai configuration, then Akamai advises setting the max reconnects to 1 instead of 0.

Allowing IPs

To allow crawling requests to cache your website without any issue, we recommend you allow Botify's user agents and/or IP addresses.

UAs (can be changed): We recommend adding a security header to secure requests, which we can create and share with you.

Cloudflare Worker IPs and UA

To allow fallback requests to access your website in case a page is not in Speedworkers cache, we recommend you allow our UA and/or IPs.


​UA (can be changed): We recommend adding a security header to secure requests, which we can create and share with you.

Testing

Use the following Curl requests to confirm the routing is working properly:

"Always Success" Test

The "always success" test will force SpeedWorkers to return a cache hit even if the page is not in the cache. This test ensures that SpeedWorkers is called and its reply returned to the bot:

Always Success (force a cache hit in SW)
--------------
URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-success,echo-67674
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: "Success"
Headers:
X-Ftlcdn-Status: false
X-Sw-Echo: 67674
X-Sw-Passed-Through: true
X-Sw-Status: success

"Cache Miss" Test

The "cache miss" test forces SpeedWorkers to return a cache miss even if it has the page in the cache. This test ensures that when SpeedWorkers can't deliver the page, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-notfound,echo-41521
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: your homepage
Headers:
X-Sw-Adn: success

"Timeout" Test

The "timeout" test forces SpeedWorkers to delay its response enough to trigger the timeout in your environment. This test ensures that when SpeedWorkers doesn't reply, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-timeout,echo-42300
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response after several seconds:
Status: 200
Body: your homepage
Headers:
X-Sw-Adn: success

Revisit Log Files

Please revisit the log files being passed to Botify for ingestion.

🛑 Botify will also conduct testing and confirm when successful. Only after receiving confirmation from Botify, proceed to Setting up Routing for Production.


Setting up Routing for Production

To be production-ready, add the remaining user agents for the bots allowed in your Botify subscription plan.

  1. In the Akamai Property Manager, navigate to Property Configuration Settings.

  2. Open the “Conditional Origin Definition” and edit the default rule to add user agents. Refer to Botify's Supported Bots list where you can copy the full list of available user agents to paste into your Akamai configuration.
    ​

  3. Save the configuration.

👉 Include wildcards when identifying user agents. Copy and paste items individually; otherwise, it will create a single match.

Troubleshooting

If sending a request to SpeedWorkers does not return the expected response when testing the integration, try the following:

❗️ Before testing with a third-party service, change the website ID and token in the recv snippet to avoid leaking them.


​
​

Did this answer your question?