Skip to main content
All CollectionsBotify ActivationSpeedWorkersSpeedWorkers Tech Docs
Deploying SpeedWorkers with Fastly
Deploying SpeedWorkers with Fastly
Updated over 6 months ago

🛠 This document explains the configuration requirements for running SpeedWorkers with Fastly.

How SpeedWorkers Works

SpeedWorkers is a service designed to deliver web pages to search engine crawler bots as fast as possible. It increases your crawl budget (SEO), serving more pages to search engines to index for the same amount of time spent on your website.

SpeedWorkers can prerender your JavaScript pages in advance, at scale, and deliver them to search engines in a few hundred milliseconds. Prerendering pages enables search engines to index your pages faster - in other words, it increases your crawl/ render budget. By storing all pages in its cache, SpeedWorkers delivers long-tail pages as fast as any other page, whereas usual CDNs cannot keep long-tail pages in the cache.

Our service has advanced quality controls to ensure the pages are rendered with all their components.

Deploy with Fastly

Step 1 - Add a new host

  1. Go to Origins > Hosts.

  2. Complete the following fields:

  • Name: SW

  • Address: Provided service delivery URL (for this example: demo.speedworkers.com)

  • Certificate hostname: Provided service delivery domain (e.g., xxx.speedworkers.com).

  • SNI hostname: Provided service delivery URL

Step 2 - Fill Override Host

  1. Click on Advanced options.

  2. Add the override host name in the Override host field.

Step 3 - Create VCL Rules

  1. ​Click on “Show VCL” in the header of the Settings section.

  2. Get the identifier of your main backend. In this example:

    the backend identifier is:

F_addr_mybucket_us_east_1_amazonaws_com

Step 4 - Create Snippets

In the final step, you create three snippets:

Create the recv snippet

  1. Click the VCL snippets link in the Settings section menu:

  2. Click Create a snippet.

  3. Fill in the following fields:

    • Name: SW recv

    • Type: recv

    • VCL:

    • replace "https://www.mysite.com" with your domain (Line 11).

    • replace "XXXXXX" with your website ID (Line 11).

    • replace "your_token" with your token (Line 11).

    • replace "XXXXXX" with your website ID (Line 14).

    • replace "your_token" with your token (Line 14).

    • Replace "F_my_origin_backend" with your origin backend you found in Step 3 (Line 17).

if((req.method == "GET" || req.method == "HEAD") && req.http.User-Agent ~ "(?i)botify-bot-sw-test" && req.url !~ "\.(js|css|woff|gif|jpe|jpg|jpeg|png|tiff|tif|pgm|xbm|pnm|ief|pbm|ras|rgb|ppm|xwd|bmp|xpm|mp2|mpeg|m1v|mpa|mpe|mp3|mpg|ppt|pps|ppt|pps|mov|doc|wav|vcf|txt|xml|ps|xml|json|qt|tsv|csv|xpdl|xls|swf|mp4|docx|eot|ico|less|rdf|rss|svg|xlm|woff|ttf|woff2|ogv|webm|pdf|map|otf|webp)$" && req.restarts == 0){
   set req.backend = SW;
   set req.http.initialURL = req.url;
   if(req.http.if-modified-since) {
     set req.http.x-sw-if-modified-since = req.http.if-modified-since;
   }
   set req.http.x-sw-client-ip = client.ip;
   set req.http.x-sw-url = "https://www.mywebsite.com" req.url;
   set req.url = "/page?website_id=XXXXXX&token=your_token";

   return(pass); // "pass" won't cache the page on fastly
 } else if (req.url ~ "^/page\?website_id=XXXXXX&token=your_token") {

 // According to Fastly the backend is not preserved after a restart, preventing the use of a condition like (req.backend == SW && req.restarts > 0) set req.url = req.http.initialURL; set req.backend = F_my_origin_backend;

 }
 return(lookup);

What does that mean?

If the user agent contains botify-sw-test-bot and:

  • ​It is the first try, use SpeedWorkers backend.

  • It is the second try, it means that SW didn’t catch the page; retry to fetch on your initial server.

Create the Fetch snippet

  1. Click Create a snippet.

  2. Fill in the following fields:

  • Name: SW fetch

  • Type: fetch

  • VCL:

if(req.backend == SW) { if(beresp.http.x-sw-status == "success") { set beresp.http.cache-control = "private"; return(deliver); } else { restart; } } return(deliver);

What does that mean?
​​If “x-sw-status” is set and returned fail, it means that SpeedWorkers can’t deliver the page, and we should not return its content. We call “restart” and the recv step is called again (falling back to your real host).

Otherwise, it means that we correctly answered, so we should deliver the page and push a “cache-control” variable to “private” to tell the CDN it should not cache the page (as we are working with long tail pages, mostly refresh by google and search engines one time a week, we give the responsibility to Botify to return them as fast as possible).

Create an Error Snippet

  1. Click Create a snippet.

  2. Fill in the following fields:

    • Name: SW error

    • Type: error

    • VCL: if(req.backend == SW) { restart; }

  3. Click Activate.

Interception

User agents to intercept:

"(?i)googlebot\/|bingbot\/|yandexbot\/|yandexmobilebot\/|baiduspider\/|baiduspider\+|applebot\/|yeti\/|botify-bot-sw-|Google-InspectionTool\/|Google-Other"

Test

To test the integration, send a request like this one (set your website ID and the URL to test):

curl -v -A "botify-bot-sw-test" -H "X-Sw-Options-Auth: WEBSITEID" -H "X-Sw-Options: always-success" https://www.mywebsite.com/ --output -

The always-success option makes SW return the request as successful even if it doesn’t have the page in cache. The response should contain x-sw-status: success in the headers and Success in the body. If your staging requires the domain to be resolved to a specific IP, you can set it in the curl command like this:

curl -v -A "botify-bot-sw-test" -H "X-Sw-Options-Auth: WEBSITEID" -H "X-Sw-Options: always-success" --resolve www.mywebsite.com:1.2.3.4 https://www.mywebsite.com/ --output -

Validating the SpeedWorkers Integration

To validate the integration of SpeedWorkers in your environment, you can send the following requests.

"Always Success" Test

The "always success" test will force SpeedWorkers to return a cache hit even if the page is not in the cache. This test ensures that SpeedWorkers is called and its reply returned to the bot:

Always Success (force a cache hit in SW)
--------------
URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-success,echo-67674
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: "Success"
Headers:
X-Ftlcdn-Status: false
X-Sw-Echo: 67674
X-Sw-Passed-Through: true
X-Sw-Status: success

"Cache Miss" Test

The "cache miss" test forces SpeedWorkers to return a cache miss even if it has the page in the cache. This test ensures that when SpeedWorkers can't deliver the page, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-notfound,echo-41521
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers

"Timeout" Test

The "timeout" test forces SpeedWorkers to delay its response enough to trigger the timeout in your environment. This test ensures that when SpeedWorkers doesn't reply, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-timeout,echo-42300
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response after several seconds:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers

Revisit Log Files

Please revisit the log files being passed to Botify for ingestion and confirm you are sending us the following:

  • origin.speedworker.com

  • cluster.speedworker.com

Troubleshooting

If sending a request to SpeedWorkers doesn’t return the expected response when testing the integration, try the following:

❗️Before testing with a third-party service, change the website ID and token in the recv snippet to avoid leaking them.

Did this answer your question?