🛠 This document explains the configuration requirements for running SpeedWorkers with NGINX.

How SpeedWorkers Works

SpeedWorkers is a service designed to deliver web pages to search engine crawler bots as fast as possible. It increases your crawl budget (SEO), serving more pages to search engines to index for the same amount of time spent on your website.

SpeedWorkers can prerender your JavaScript pages in advance, at scale, and deliver them to search engines in a few hundred milliseconds. Prerendering pages enables search engines to index your pages faster - in other words, it increases your crawl/ render budget. By storing all pages in its cache, SpeedWorkers delivers long-tail pages as fast as any other page, whereas usual CDNs cannot keep long-tail pages in the cache.

Our service has advanced quality controls to ensure the pages are rendered with all their components.

Configuring NGINX

The following describes how to configure your NGINX environment for SpeedWorkers. Configuring NGINX for SpeedWorkers includes the following steps:

Add the SpeedWorkers configuration to your NGINX configuration
Update SpeedWorkers configuration with the values we provide
Ensure that your configuration is compatible

1: Add the SpeedWorkers configuration to your NGINX configuration

Add the code between the BEGINNING and END comments to your NGINX configuration. It should be placed high enough so that it is not impacted by any transformations you may make to the original request. Other parts of the configuration (listen, location /) are kept only as an example of a complete working configuration.

server {
  listen 80 default_server;
  listen [::]:80 default_server;
  server_name _;

  ###########################################
  # BEGINNING OF SPEEDWORKERS CONFIGURATION #
  ###########################################

  # 1. Place this configuration needs high in within your server directive.
  # It ensures that SpeedWorkers intercepts requests before any
  # transformations you may perform (rewrites…).
  #
  # 2. Update this configuration with the following values we provide:
  # - `Host` header
  # - `X-Sw-Adn-Token` header
  # - `proxy_pass` directive
  #
  # 3. Make sure that all headers are passed as is.
  #
  # 4. Make sure that SpeedWorkers requests are NOT cached.
  #
  # 5. You may change the `__speedworkers__` location and rewrite to any 
  # private value.

  # Default to no interception
  set $sw_should_intercept 0;

  # Intercept only requests from known bots
  if ($http_user_agent ~* "googlebot/|bingbot/|yandexbot/|yandexmobilebot/|baiduspider/|baiduspider+|applebot|yeti|google-inspectiontool/|googleother|botify-bot-sw") { set $sw_should_intercept 1;
  }
  # Do not intercept requests for static assets
  # /!\ Remove any of the following file extensions if it matches your tld. if ($uri ~* "\.(js|css|map|xml|less|png|jpg|jpeg|gif|tiff|pdf|doc|txt|ico|rss|zip|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp3|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|otf|woff|woff2|svg|eot)$") { set $sw_should_intercept 0;
  }

  # Intercept only GET and HEAD requests
  # If you use non-standard HTTP methods, SpeedWorkers will handle them
  # but it is more efficient to add them here.
  if ($request_method = POST) { set $sw_should_intercept 0; }
  if ($request_method = PUT) { set $sw_should_intercept 0; }
  if ($request_method = PATCH) { set $sw_should_intercept 0; }
  if ($request_method = DELETE) { set $sw_should_intercept 0; }
  if ($request_method = OPTIONS) { set $sw_should_intercept 0; }
  if ($request_method = CONNECT) { set $sw_should_intercept 0; }
  if ($request_method = TRACE) { set $sw_should_intercept 0; }

  # Do not intercept WebSocket requests
  if ($http_upgrade = "websocket") { set $sw_should_intercept 0; }

  # Perform interception
  if ($sw_should_intercept) {
    # Rewrite to force NGINX to match the SpeedWorkers location
    rewrite ^.+$ /__speedworkers__ last;
  }

  # Send intercepted requests to SpeedWorkers
  # This location should only be accessed by the above rewrite
  location = /__speedworkers__ {
    # Cleanup the url sent to the worker
    set $args '';
    rewrite ^.+$ / break;

  # Resolve DNS through cloudflare service
  resolver 1.1.1.1;

  proxy_connect_timeout 10s;
  proxy_read_timeout 10s;
  proxy_ssl_server_name on;

  proxy_set_header X-Sw-Url $scheme://$host$request_uri;

  # TODO: Update with the value we provide
  proxy_set_header X-Sw-Adn-Token MY_SW_ADN_TOKEN;

  # TODO: Update with the value we provide
  proxy_set_header Host worker;

  # /!\ proxy_pass must be the last directive

  # TODO: Update with the url we provide
  proxy_pass http://worker:3000;
 }

  #####################################
  # END OF SPEEDWORKERS CONFIGURATION #
  #####################################
 
 # Handle other requests normally with your own configuration
 # Your configuration is not impacted by SpeedWorkers and the following
 # configuration is just an example.
 location / {
 proxy_pass http://origin:3001;
 proxy_http_version 1.1;
 proxy_set_header Host $http_host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "Upgrade";
 }
}

2: Update SpeedWorkers configuration with the values we provide

Every value to update is marked with a TODO: comment in the above example.

Update the Host header with the worker's hostname (e.g., proxy_set_header Host cqj7pktm.sw.adn-test.cloud;).
Update the X-Sw-Adn-Token.
Update the proxy_pass directive with the worker’s URL (e.g., proxy_pass https://cqj7pktm.sw.adn-test.cloud;).

3: Ensure that your configuration is compatible

To work properly, you need to ensure all of the following:

The requested URL is passed as-is.
All headers are passed as-is.
SpeedWorkers requests are not cached.

Validating the SpeedWorkers Integration

To validate the integration of SpeedWorkers in your environment, you can send the following requests.

"Always Success" Test

The "always success" test will force SpeedWorkers to return a cache hit even if the page is not in the cache. This test ensures that SpeedWorkers is called and its reply returned to the bot:

Always Success (force a cache hit in SW)
--------------
URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-success,echo-67674
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: "Success"
Headers:
X-Ftlcdn-Status: false
X-Sw-Echo: 67674
X-Sw-Passed-Through: true
X-Sw-Status: success

"Cache Miss" Test

The "cache miss" test forces SpeedWorkers to return a cache miss even if it has the page in the cache. This test ensures that when SpeedWorkers can't deliver the page, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-notfound,echo-41521
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers

"Timeout" Test

The "timeout" test forces SpeedWorkers to delay its response enough to trigger the timeout in your environment. This test ensures that when SpeedWorkers doesn't reply, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-timeout,echo-42300
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response after several seconds:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers

Revisit Log Files

Please revisit the log files being passed to Botify for ingestion and confirm you are sending us the following:

origin.speedworker.com
cluster.speedworker.com

Troubleshooting

If sending a request to SpeedWorkers doesn’t return the expected response when testing the integration, try the following:

Replace the SpeedWorkers host (origin) with a third-party service like PutsReq, Request Catcher - record HTTPS requests, webhooks, API calls, or Beeceptor - Rest API mocking in seconds. It will help you verify that the call to SW is correct.
Remove the failover rule to get details in the edge diagnostic tools about why the request to SpeedWorkers fails, such as the Akamai reference error.

❗️Before testing with a third-party service, change the website ID and token in the recv snippet to avoid leaking them.