🛠 This document explains the configuration requirements for running SpeedWorkers with Cloudflare.
How SpeedWorkers Works
SpeedWorkers is a service designed to deliver web pages to search engine crawler bots as fast as possible. It increases your crawl budget (SEO), serving more pages to search engines to index for the same amount of time spent on your website.
SpeedWorkers can prerender your JavaScript pages in advance, at scale, and deliver them to search engines in a few hundred milliseconds. Prerendering pages enables search engines to index your pages faster - in other words, it increases your crawl/ render budget. By storing all pages in its cache, SpeedWorkers delivers long-tail pages as fast as any other page, whereas usual CDNs cannot keep long-tail pages in the cache.
Our service has advanced quality controls to ensure the pages are rendered with all their components.
About Cloudflare Workers
Cloudflare triggers Worker functions before processing requests.
The Cloudflare worker created for SpeedWorkers calls the SpeedWorkers API and either returns the cached page or lets Cloudflare continue its regular process.
How to install it on Cloudflare Workers?
Connect to your Cloudflare console
👉 Your Cloudflare subscription plan must include access to Cloudflare Workers.
Go to the "Workers Routes" section.
Click the Enable Workers button, then click Manage Workers:
Click the Create application button:
Click the Create Worker button:
Select a worker name, then click Deploy:
Click Edit code.
Worker Configuration:
Delete existing lines.
Paste the SpeedWorkers code (see below in Configuration).
Set the configuration (website ID, authentication token, SpeedWorkers URL). Don’t forget the “/page?” at the end of the “swUrl”.
Click on Save and Deploy to save the worker.
Delete existing lines
👉 You can test the worker in the preview panel. "Cache miss" messages display in the console if you request pages that SpeedWorkers has not yet indexed and cached.
Configuration
Copy/Paste the SpeedWorkers Code from this gist: Cloudflare Worker
After pasting the SpeedWorkers code in the worker editor, you may have to modify some settings:
Name | Status | Role |
swClientWebsiteId | REQUIRED | Your website ID to authenticate requests |
swClientToken | REQUIRED | Your token to authenticate requests |
swUrl | REQUIRED | The URL to the SW API |
swRewriteOrigin | OPTIONAL | If there is a difference between the domain in the URLs indexed and the domain in URLs crawled by the search engine bots, use this setting to replace the domain in the request sent to the SWBeam API |
swRestrictToUrls | OPTIONAL | Instead of creating Cloudflare routes, you can specify the paths SW should intercept. |
swProgressiveRolloutPm | REQUIRED | Set a value < 1000 if you want only a per mille of the pages SW handles. |
❗️Do not modify other Worker settings.
👉 To test the implementation before going live with search engine bots, you can modify the bot pattern and set it to: const botPattern = "(botify-bot-sw-)";
Routing
You have to create a route on which the worker will be called.
Return to the “Workers Routes” section of the Cloudflare Overview page, then click Manage Workers:
Select your new Worker from the list.
In the Triggers section, click Add route.
Add your domain/subdomain on the “route” section and select the zone related to this domain/subdomain. Then click the Add route button.
Disable the default route by clicking on "..." .
💡 To limit SpeedWorkers to specific domains, specify these domains directly in your Cloudflare worker configuration or create different Cloudflare routes.
Caching
The Cloudflare worker returns a "cache-control" header set to "max-age 0" for pages from the SpeedWorkers cache. It means Cloudflare will not cache these pages. It ensures Cloudflare will not deliver SpeedWorkers pages to end-users. Only search engine bots should get those pages.
Validating the SpeedWorkers Integration
To validate the integration of SpeedWorkers in your environment, you can send the following requests.
"Always Success" Test
The "always success" test will force SpeedWorkers to return a cache hit even if the page is not in the cache. This test ensures that SpeedWorkers is called and its reply returned to the bot:
Always Success (force a cache hit in SW)
--------------
URL: Your homepage (https://www.mywebsite.com)
Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-success,echo-67674
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify
Expected response:
Status: 200
Body: "Success"
Headers:
X-Ftlcdn-Status: false
X-Sw-Echo: 67674
X-Sw-Passed-Through: true
X-Sw-Status: success
"Cache Miss" Test
The "cache miss" test forces SpeedWorkers to return a cache miss even if it has the page in the cache. This test ensures that when SpeedWorkers can't deliver the page, the request falls back properly:
URL: Your homepage (https://www.mywebsite.com)
Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-notfound,echo-41521
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify
Expected response:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers
"Timeout" Test
The "timeout" test forces SpeedWorkers to delay its response enough to trigger the timeout in your environment. This test ensures that when SpeedWorkers doesn't reply, the request falls back properly:
URL: Your homepage (https://www.mywebsite.com)
Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-timeout,echo-42300
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify
Expected response after several seconds:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers
Revisit Log Files
Please revisit the log files being passed to Botify for ingestion and confirm you are sending us the following:
origin.speedworker.com
cluster.speedworker.com
Troubleshooting
If sending a request to SpeedWorkers doesn’t return the expected response when testing the integration, try the following:
Replace the SpeedWorkers host (origin) with a third-party service like PutsReq, Request Catcher - record HTTPS requests, webhooks, API calls, or Beeceptor - Rest API mocking in seconds. It will help you verify that the call to SW is correct.
Remove the failover rule to get details in the edge diagnostic tools about why the request to SpeedWorkers fails, such as the Akamai reference error.
❗️Before testing with a third-party service, change the website ID and token in the recv snippet to avoid leaking them.