🛠 This document explains the configuration requirements for running SpeedWorkers with Fastly in our recommended implementation.

How SpeedWorkers Works

SpeedWorkers is a service designed to deliver web pages to search engine crawler bots as fast as possible. It increases your crawl budget (SEO), serving more pages to search engines to index for the same amount of time spent on your website.

SpeedWorkers can prerender your JavaScript pages in advance, at scale, and deliver them to search engines in a few hundred milliseconds. Prerendering pages enables search engines to index your pages faster - in other words, it increases your crawl/ render budget. By storing all pages in its cache, SpeedWorkers delivers long-tail pages as fast as any other page, whereas usual CDNs cannot keep long-tail pages in the cache.

Our service has advanced quality controls to ensure the pages are rendered with all their components.

Deploying with Fastly

The following describes how to configure your environment for the Recommended SpeedWorkers implementation, where SpeedWorkers will retrieve pages not in the cache from your origin server. To configure your environment to handle this fallback on your own, please refer to the Advanced Implementation guide.

Configuring Fastly for the recommended SpeedWorkers implementation includes the following steps:

Adding a new host
Creating VCL rules

1: Add a New Host

Navigate to Origins > Hosts.
Complete the fields as follows:
- Name: SW
- Address: Provided service delivery URL (e.g., demo.speedworkers.com)
- Certificate hostname: Provided service delivery URL (e.g., demo.speedworkers.com)
- SNI hostname: Provided service delivery URL (e.g., demo.speedworkers.com)
- Override Host: Provided service delivery URL (e.g., demo.speedworkers.com)
Optionally set the advanced options as follows:
- Maximum connections: The maximum number of connections to your origin per cache node. The default is 200.
- Error threshold: The number of 503 errors to allow before the backend status is considered down.
- Connection timeout: The number of milliseconds to wait for a timeout. The default is 1,000 ms.
- First byte timeout: The number of milliseconds to wait for a first-byte timeout. The default is 15,000 milliseconds.
- Between bytes timeout: The number of milliseconds to wait between bytes. The default is 10,000 milliseconds.

2: Create VCL Rules

Navigate to VCL snippets.
Click the Create snippet button.
Complete the fields as follows:
- Name: SW recv
- Type: recv (vcl_recv)

Copy/Paste the code below into the VCL field, changing the following:

Replace the “ADNTOKEN" value with your ADN Token provided by Botify (don't forget the quotes).

Replace the "http://Mywebsite.com" value with your website URL (don't forget the quotes).

if((req.method == "GET" || req.method == "HEAD") && req.http.User-Agent ~ "(?i)botify-bot-sw-test" && req.url !~ "\.(js|css|woff|gif|jpe|jpg|jpeg|png|tiff|tif|pgm|xbm|pnm|ief|pbm|ras|rgb|ppm|xwd|bmp|xpm|mp2|mpeg|m1v|mpa|mpe|mp3|mpg|ppt|pps|ppt|pps|mov|doc|wav|vcf|txt|xml|ps|xml|json|qt|tsv|csv|xpdl|xls|swf|mp4|docx|eot|ico|less|rdf|rss|svg|xlm|woff|ttf|woff2|ogv|webm|pdf|map|otf|webp)$" && req.restarts == 0) {
 set req.backend = SW;
 set req.http.initialURL = req.url;
 if(req.http.if-modified-since) {
   set req.http.x-sw-if-modified-since = req.http.if-modified-since;
 }
 set req.http.x-sw-client-ip = client.ip;
     set req.http.x-sw-adn-token = "ADNTOKEN"; //Modify with your ADN Token
 set req.http.x-sw-url = "http://Mywebsite.com" req.url; //Modify with your website Url return(pass); // "pass" won't cache the page on Fastly
 }
 return(lookup);

This code means if the user agent contains botify-sw-test-bot and it is the first try, it will use the SpeedWorkers backend created in Step 1.

Click the Create button to validate the creation of the snippet:
Click the Activate button to activate the new version of your configuration:
After Botify validates the previous steps:
- Update the VCL snippet created in Step 2:4 to route traffic for search engine bots.
- Change the value (?i)botify-bot-sw-test to (?i)bingbot\/|googlebot/\|botify-bot-sw-test (this value may vary depending on the contract).

Validating the SpeedWorkers Integration

To validate the integration of SpeedWorkers in your environment, you can send the following requests.

"Always Success" Test

The "always success" test will force SpeedWorkers to return a cache hit even if the page is not in the cache. This test ensures that SpeedWorkers is called and its reply returned to the bot:

Always Success (force a cache hit in SW)
--------------
URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-success,echo-67674
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: "Success"
Headers:
X-Ftlcdn-Status: false
X-Sw-Echo: 67674
X-Sw-Passed-Through: true
X-Sw-Status: success

"Cache Miss" Test

The "cache miss" test forces SpeedWorkers to return a cache miss even if it has the page in the cache. This test ensures that when SpeedWorkers can't deliver the page, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-notfound,echo-41521
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: your homepage
Headers:
X-Sw-Adn: success

"Timeout" Test

The "timeout" test forces SpeedWorkers to delay its response enough to trigger the timeout in your environment. This test ensures that when SpeedWorkers doesn't reply, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-timeout,echo-42300
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response after several seconds:
Status: 200
Body: your homepage
Headers:
X-Sw-Adn: success

Revisit Log Files

Please revisit the log files being passed to Botify for ingestion and confirm you are sending us the following:

origin.speedworker.com
cluster.speedworker.com

Troubleshooting

If sending a request to SpeedWorkers doesn’t return the expected response when testing the integration, try the following:

Replace the SpeedWorkers host (origin) with a third-party service like PutsReq, Request Catcher - record HTTPS requests, webhooks, API calls, or Beeceptor - Rest API mocking in seconds. It will help you verify that the call to SW is correct.
Remove the failover rule to get details in the edge diagnostic tools about why the request to SpeedWorkers fails, such as the Akamai reference error.

❗️Before testing with a third-party service, change the website ID and token in the recv snippet to avoid leaking them.