🛠 This document explains the configuration requirements for running SpeedWorkers with HAProxy.

Download the SpeedWorkers HAProxy LUA script

The HAProxy LUA script is available here. Download the files and put them on your HAProxy server.

Configuring HAProxy

Add this to your HAProxy configuration file (/etc/haproxy/haproxy.cfg):

global ... # load SpeedWorkers script lua-load <PATH_TO_LUA_FILES>/sw.lua ... frontend http_front ... # SpeedWorkers is only interested by bot requests for web pages (not for static resources) acl is_bot hdr_reg(user-agent) -i (googlebot/|bingbot/|yandexbot/|yandexmobilebot/|baiduspider/|baiduspider+|applebot/|yeti/|googleother|google-inspectiontool/|botify-bot-sw-) acl is_resource path_reg -i .\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|ief|jpe|jpeg|jpg|js|json|less|map|m1v|mka|mkv|mov|mp2|mp3|mp4|mpa|mpe|mpeg|mpg|odt|otf|ott|ogg|ogm|ogv|opus|pbm|pdf|pgm|png|ppm|pps|ppt|pptx|ps|qt)$ # Set some variables useful for SpeedWorkers http-request set-var(txn.sw.protocol) lua.protocol() http-request set-var(txn.sw.url) url http-request set-var(txn.sw.host) hdr(host) http-request set-var(txn.sw.sni) ssl_fc_sni http-request set-var(txn.sw.client_ip) src http-request set-var(txn.sw.ims) hdr(if-modified-since) # Call SpeedWorkers if it's a bot request for a web page and returned the cached page if the request is successful http-request lua.sw-request 8082 if is_bot !is_resource http-request use-service lua.deliver_sw_response if { var(txn.sw.success) -m bool } ... # You can use you own resolver in the speedworkers_backend instead resolvers mydns nameserver dns1 1.1.1.1:53 accepted_payload_size 8192 # allow larger DNS payloads resolve_retries 30 timeout retry 1s hold valid 10s # Bind a local port to let the LUA call SpeedWorkers in TLS! frontend speedworkers_proxy mode http # If you change the port, change it in the *frontend* as well bind 127.0.0.1:8082 default_backend speedworkers_backend # Change the XXX with the name provided by Botify backend speedworkers_backend mode http server speedworkers XXX.speedworkers.com:443 check-ssl check-sni XXX.speedworkers.com ssl sni str(XXX.speedworkers.com) verify required ca-file ca-certificates.crt resolvers mydns resolve-prefer ipv4

Configuration Customization

Replace <PATH_TO_LUA_FILES> with the path to the LUA script files, or use the lua-prepend-path keyword
Replace all occurrences of XXX.speedworkers.com with the SpeedWorkers domain provided by Botify
This configuration binds a local port (8082). You can change it with another port but make sure to change it in the bind 127.0.0.1:8082 directive AND in the http-request lua.sw-request 8082 directive

LUA Script Customization

Set the SPEEDWORKERS_WEBSITE_ID variable to the value provided by Botify
Set the SPEEDWORKERS_TOKEN to the value provided by Botify
Set the SPEEDWORKERS_DOMAIN to the value provided by Botify
Replace the <PATH_TO_LUA_FILES> with the correct path

Tests

To test the integration before going live, you can replace the is_bot ACL with this one:

acl is_bot hdr_reg(user-agent) -i (botify-bot-sw-)

With this, only the request with the botify-bot-sw-… user-agent will be intercepted by SpeedWorkers.

Validating the SpeedWorkers Integration

To validate the integration of SpeedWorkers in your environment, you can send the following requests.

"Always Success" Test

The "always success" test will force SpeedWorkers to return a cache hit even if the page is not in the cache. This test ensures that SpeedWorkers is called and its reply returned to the bot:

Always Success (force a cache hit in SW)
--------------
URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-success,echo-67674
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: "Success"
Headers:
X-Ftlcdn-Status: false
X-Sw-Echo: 67674
X-Sw-Passed-Through: true
X-Sw-Status: success

"Cache Miss" Test

The "cache miss" test forces SpeedWorkers to return a cache miss even if it has the page in the cache. This test ensures that when SpeedWorkers can't deliver the page, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-notfound,echo-41521
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers

"Timeout" Test

The "timeout" test forces SpeedWorkers to delay its response enough to trigger the timeout in your environment. This test ensures that when SpeedWorkers doesn't reply, the request falls back properly:

URL: Your homepage (https://www.mywebsite.com)

Headers:
User-Agent: botify-bot-sw-test
X-Sw-Options: passed-through,request-time,always-timeout,echo-42300
X-Sw-Options-Auth: XXXXXX <= the website ID provided by Botify

Expected response after several seconds:
Status: 200
Body: your homepage
Headers:
NO X-Sw-... headers

Troubleshooting

When testing the integration, if sending a request to SW doesn’t return the expected response, try replacing the SW host (origin) with a third-party service like PutsReq, Request Catcher — record HTTP requests, webhooks, API calls, or Beeceptor - Rest API mocking in seconds. It will help you verify that the call to SW is correct.

❗️Before testing with a third-party service, change the website ID and token in the recv snippet to avoid leaking them.