📘 This is a step-by-step guide to routing search engine and AI bot traffic to SpeedWorkers using a single Amazon CloudFront Function, while your users continue to be served from your existing origin.

How it works

SpeedWorkers pre-renders your pages in advance and delivers them to crawlers in a few hundred milliseconds, increasing the crawl and render budget search engines spend on your site. To send bot traffic to SpeedWorkers, CloudFront needs to identify each request and route it to the right origin. This is handled by a single CloudFront Function attached to the viewer request event of your behaviors. For every incoming request it:

Detects whether the request comes from a bot or a human user.
Adds the X-Sw-* headers SpeedWorkers needs (request type, host, user-agent, and so on).
Routes bot requests to the SpeedWorkers origin, and leaves human requests on your default origin untouched.

This CloudFront Function approach replaces the older two-Lambda@Edge setup. It is simpler to deploy and significantly cheaper to run, because the function executes at the edge without invoking Lambda.

How the CloudFront function routes requests

The function runs on the viewer request event and decides, for every request, where CloudFront should send it. Its behavior is driven by the three values you set in Step 2 — SW_ORIGIN_ID, SW_ORIGIN_DOMAIN and DEFAULT_ORIGIN_ID.

Method & path filtering. Only GET / HEAD page requests are considered. Other methods, static assets and API paths (/css/, /js/, images, .pdf, and so on) always pass straight through to your website.
Bot detection. The request's User-Agent is matched against the bot pattern. During validation this is limited to botify-bot-sw-; the production pattern covers the major search engine and AI crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, and more).
Human visitors. Requests that don't match are tagged x-sw-request-type: user and served from your website (DEFAULT_ORIGIN_ID) unchanged — no added latency.
Bots. Matching requests get the SpeedWorkers headers added — x-sw-host, x-sw-user-agent, x-sw-client-ip, x-sw-uri (the full rebuilt URL), x-sw-if-modified-since when present, and a unique x-sw-request-type.
Routing & failover. The function points the request at an origin group it builds on the fly: SpeedWorkers as primary (SW_ORIGIN_ID / SW_ORIGIN_DOMAIN) and your website as failover (DEFAULT_ORIGIN_ID). On a cache hit, SpeedWorkers returns its stored response. On a cache miss it returns a 403, or a 502 if it is down — any failover status code (400, 403, 416, 429, 500, 502, 503, 504) makes CloudFront serve the request from your website instead.

You don't need to create the origin group yourself since the function builds it per request. Your website ID and token are never placed in the function; they live as custom headers on the SpeedWorkers origin (Step 1).

⚠️ A 403 from SpeedWorkers is normal — it is the cache-miss signal that triggers failover to your origin. For failover to work on every request, CloudFront must not cache error responses (see Step 5). A cached 403 would freeze the failover state and break normal delivery. This design is also fail-safe: if the function is ever removed, SpeedWorkers is simply no longer called and every request is served from your origin as usual.

Before you begin

Botify will provide the following token and website ID values to authenticate your traffic to SpeedWorkers. Keep them secure.

Value	Provided by
SpeedWorkers origin domain `*.speedworkers.com`	Botify
Website ID `x-sw-websiteid`	Botify
Token `x-sw-token`	Botify
Your existing website origin ID	Your CloudFront distribution

You will also need access to the AWS CloudFront console with permission to create origins, CloudFront Functions, and cache / origin request policies.

Step 1: Create the SpeedWorkers origin

AWS Console › CloudFront › Distributions › [Your distribution] › Origins

Click Create origin.
Set Origin domain to the SpeedWorkers domain provided by Botify.
Set Protocol to HTTPS only.
Set Name to SW-origin.
Set the additional settings.
Click Create origin.

Add two custom headers

In the Add custom header section, add the following values, which are provided by Botify.

Header name	Value
`x-sw-websiteid`	Website ID provided by Botify
`x-sw-token`	Token provided by Botify

Additional settings

Expand Additional settings and set the timeouts so requests fall back cleanly if SpeedWorkers is slow to respond.

Step 2: Create the CloudFront function

AWS Console › CloudFront › Functions

Click Create function and give it a name (e.g., speedworkers-routing).
Select runtime cloudfront-js-2.0.
Replace the default code with the function provided by Botify: gist.github.com/jblernout/366a7a356c363263f604d902dfd7913e.
Update the three values at the top of the file to match your setup, then Save changes and Publish.

// To be updated
var SW_ORIGIN_ID     = "SW-origin"; // SpeedWorkers origin ID — must match the Name from Step 1
var SW_ORIGIN_DOMAIN = "YourSWdomain";  // SpeedWorkers domain provided by Botify
var DEFAULT_ORIGIN_ID = "website-origin-ID"; // Your existing website origin ID
// End of update

Variable	Set to
`SW_ORIGIN_ID`	The origin Name you created in Step 1 (SW-origin).
`SW_ORIGIN_DOMAIN`	The SpeedWorkers domain provided by Botify.
`DEFAULT_ORIGIN_ID`	The ID of your existing website origin.

Step 3: Associate the Function with your behaviors

AWS Console › CloudFront › Distributions › [Your distribution] › Behaviors

Attach the function to each behavior that should route to SpeedWorkers — typically your default behavior and any other HTML page behaviors.

Select the behavior and click Edit.
Under Function associations, set Viewer request to the speedworkers-routing CloudFront Function.
Click Save changes.
Repeat for every behavior that serves pages SpeedWorkers should handle.

⚠️ Only associate the function with behaviors that serve HTML pages. Attaching it to behaviors for static assets (images, JS, CSS) or APIs adds unnecessary edge execution cost with no benefit.

Step 4: Configure your cache and origin request policies

AWS Console › CloudFront › Distributions › [Your distribution] › Behaviors › Edit

A. Partition the cache between users and bots

Edit your cache policy so the bot and user versions of a page are cached separately. Otherwise, a pre-rendered bot page could be served to a real visitor.

In the Headers dropdown, select Include the following headers if it is set to None (leave it as None only if all TTL settings are 0).
Add the custom header X-Sw-Request-Type. This lets SpeedWorkers partition the cache between users and bots.
Save the changes.

B. Forward headers to the origin

Edit your origin request policy so SpeedWorkers receives the context it needs. In the Headers dropdown, either select All viewer headers, or select Include the following headers and add every header below (plus any of your own):

Header	Purpose
X-Sw-Request-Type	Bot vs. user classification
X-Sw-Host	Original request host, used to rebuild the URL
X-Sw-User-Agent	Original user-agent forwarded to SpeedWorkers
X-Sw-If-Modified-Since	Conditional-request support
X-Sw-Options	SpeedWorkers request options
X-Sw-Options-Auth	Authentication for the options (website ID)
X-Sw-uri	Original request URI
X-Sw-client-ip	Original client IP address

In the Cookies and Query strings dropdowns, keep your usual settings. Make sure query strings that influence page content are forwarded, then save.

Step 5: Disable error-page caching

AWS Console › CloudFront › Distributions › [Your distribution] › Error pages

SpeedWorkers returns a 403 on a cache miss to trigger failover to your origin. If CloudFront caches that 403, the failover response is served again and again and normal pages break. Error responses must never be cached, especially the 403.

Open the Error pages (custom error responses) tab of your distribution.
Create a custom error response for HTTP 403.
Set the Error caching minimum TTL to 0.
Repeat for the other error codes CloudFront caches by default (400, 404, 500, 502, 503, 504).

⚠️ The 403 is the most important code to set to a 0 TTL — it is SpeedWorkers' cache-miss and failover signal.

Step 6: Test and validate

SpeedWorkers exposes test requests that let you verify each part of the integration without affecting live bot traffic. Send them to your homepage and replace the website ID where noted.

During validation, use a user-agent containing botify-bot-sw- so only your test requests are routed to SpeedWorkers and real bot traffic is left untouched until you are ready to go live.

Test 1 — Always success (forces a cache hit)

Confirms SpeedWorkers is reached and its response is returned to the bot.

URL: Your homepage (https://www.mywebsite.com)
Headers:
  User-Agent:        botify-bot-sw-test
  X-Sw-Options:      passed-through,request-time,always-success,echo-67674
  X-Sw-Options-Auth: <your website ID>

Expected response:
  Status: 200
  Body:   Success
  Headers: X-Sw-Status: success · X-Sw-Echo: 67674 · X-Sw-Passed-Through: true

Test 2 — Cache miss (forces a fallback)

Confirms that when SpeedWorkers has no page to serve, the request falls back to your origin.

URL:  https://www.yourdomain.com
Headers:
  User-Agent:        botify-bot-sw-test
  X-Sw-Options:      passed-through,request-time,always-notfound,echo-41521
  X-Sw-Options-Auth: <your website ID>

Expected response:
  Status: 200
  Body:   your homepage
  Headers: no X-Sw-* headers

Test 3 — Timeout (forces a fallback)

Confirms that when SpeedWorkers does not reply in time, the request falls back to your origin.

URL:  https://www.yourdomain.com
Headers:
  User-Agent:        botify-bot-sw-test
  X-Sw-Options:      passed-through,request-time,always-timeout,echo-42300
  X-Sw-Options-Auth: <your website ID>

Expected response (after several seconds):
  Status: 200
  Body:   your homepage
  Headers: no X-Sw-* headers

Once the function is deployed in a publicly reachable environment, Botify can run an automated batch of these tests. Otherwise, run them manually.

Troubleshooting

Symptom	Likely cause	Fix
Bot requests are not served by SpeedWorkers	Function not associated, or wrong origin IDs	Confirm the function is set on Viewer request (Step 3) and that `SW_ORIGIN_ID` / `DEFAULT_ORIGIN_ID` match your origin names (Step 2).
Human visitors receive prerendered bot pages	Cache is not partitioned	Add `X-Sw-Request-Type` to the cache policy headers (Step 4A).
403 on a page that should be cached, or on the always-success test	Invalid SpeedWorkers headers (wrong website ID / token, or headers not being added)	A 403 on a normal request is expected — it is a cache miss that triggers failover. It is only a problem when it happens on a forced cache-hit test or a page you know is in SpeedWorkers' cache. Re-check `x-sw-websiteid` and `x-sw-token` on the SW origin (Step 1) and that the function is attached (Step 3).
Wrong content variant returned	Query strings not forwarded	Ensure query strings that affect content are forwarded in the origin request policy (Step 4B).
Failover not working / stale error responses served to bots	CloudFront is caching the 403 (cache miss) or 502 (SpeedWorkers down) that triggers failover	Set the Error caching minimum TTL to 0 for all error codes, especially 403 and 502 (Step 5).

👉 If a test still fails after these checks, contact your Botify representative with the request headers you sent and the full response headers you received.

Deploying SpeedWorkers with CloudFront