📘 This article provides information on how to set the cache behaviors to optimize your cache refresh in SpeedWorkers. SpeedWorkers is part of Botify's Activation Suite, available as an option with a Botify Pro or Enterprise plan.
Overview
The Cache Behavior settings in SpeedWorkers determine how the pages in your scope are fetched and rendered and how long pages remain in page inventory. By setting different cache rates and rules for different types of content, you can optimize your page inventory by allowing SpeedWorkers to render more efficiently. You can set both default cache behaviors and custom behaviors that override the default behaviors for specific pages.
Access the Cache Behavior settings by navigating to Activation > Speedworkers > Settings > Cache Behaviors:
Refresh Capacity Rate
A banner displays the current refresh capacity rate at the top of the Cache Behaviors settings page. This provides visibility into whether SpeedWorkers can refresh all pages in your inventory based on current settings. Please refer to the Managing Refresh Capacity in SpeedWorkers article for more information.
Defining the Default Behavior
The default behavior defines the cache refresh behavior for all URLs in your scope that do not match a specified custom behavior. The default behavior includes the following rules:
To define the default behavior:
If settings are not in Edit mode, refer to the Defining SpeedWorkers Settings article to enable editing.
Click the pencil icon in the Default behavior block:
Define the default rules, and then click Next.
Optionally, add advanced rules.
Optionally, choose to serve Markdown to AI bots, then click Save.
Defining default rules
The default rules define the rate and priority at which pages are refreshed, whether separate versions for mobile and desktop are cached, and how JavaScript redirections should be handled.
A JS redirection is when the JavaScript code changes the URL of the current page without making the browser reload the page. When SpeedWorkers detects a JS redirection, you can have SpeedWorkers not cache the page or cache it as a 301 or 302 redirection.
To define default rules:
Select the number and time increment to identify the rate at which pages in the SpeedWorkers inventory should be refreshed.
If an issue prevents page refresh at the above rate, the pages will enter a "stale" state. Select the number and time increment to identify how long after the latest refresh attempt to serve the stale pages. We recommend setting this at approximately 25% of the refresh rate (e.g., for a weekly refresh, serve stale pages for 2 days). This allows SpeedWorkers to continue serving the pages to bots instead of the pages being moved directly into an expired state. Learn more about page states in the Understanding the Page Lifecycle in SpeedWorkers article.
Click the Open Advanced Settings link to identify the action to occur when a JavaScript redirection is encountered on pages within the defined scope.
We recommend leaving the default "text/html" value in the Content-Type field. To change this value, click the pencil icon to enable editing and then identify the page content type.
From the JS Redirection dropdown list, select the action to occur when a JavaScript redirection is detected.
Select the priority at which these pages should be refreshed to instruct SpeedWorkers which pages to refresh first (i.e., high refresh priority pages will be refreshed first).
In the "Mobile & Desktop Versions" section:
Select the "Responsive Version" option to cache only one version of the page; this is the recommended option for sites that have responsive pages.
Select the "Desktop version/Mobile version" option to cache separate desktop and mobile page versions.
From the Mobile & Desktop rendering dropdown lists, choose whether SpeedWorkers should render pages with the default JavaScript configuration, HTML configuration, or not. The render option is the same for mobile and desktop pages when "Responsive Version" is selected and separate when "Desktop version/Mobile version" is selected.
To have SpeedWorkers serve clean URLs without parameters, identify the parameters to be ignored when SpeedWorkers caches pages. Type parameters in the text box, separating multiple parameters with commas, and then click the Add parameter link. Parameters are listed below the text box. Click the X next to a parameter to delete it from the list, or click the Clear all button at the right to remove all parameters.
If you identified parameters to be ignored, select an action from the dropdown list to identify the behavior for pages with removed parameters. Select the "Remove ignored parameters . . . " checkbox to remove the parameters identified above from all links, canonicals, and hreflang attributes on all pages within the scope.
Click Next to add advanced rules.
Defining advanced rules
Advanced Rules help you avoid refresh waste by allowing you to make rules based on HTTP code, page property, or input source. Adding these additional rules allows you to determine the behavior when a page is refreshed but the status code or page property has changed. You can create additional rules based on the following:
Current or previous HTTP status code.
Count Same Consecutive HTTP Status Code: The number of times the page returned the same status code (consecutively).
Duration Same Consecutive HTTP Status Code: How long the page has been returning the same status code, in minutes.
Current or previous page property: A page property describes the page's content or an element (e.g., a profile page or product availability). Botify can extract any value from the HTML and set it as a page property. If you are unaware of the page properties set for your project, please contact your SEO Success Manager.
Inventory input: The source from where the pages were added to the SpeedWorkers inventory.
To add an advanced rule:
Select a criterion, operator, and value to define the rule (e.g., Current HTTP Status Code equals 200).
Click the Add a condition link to add additional criteria to the rule.
Select a refresh rate, the time the page can be stale, and the refresh priority (i.e., the order in which SpeedWorkers will cache the pages).
Click the Do not update cache checkbox SpeedWorkers if you do not want the cache to update when SpeedWorkers encounters pages that fit this rule.
Optionally, click the Add an advanced rule link to add additional rules.
Click Save.
Advanced Rule Examples
You may want to increase the refresh rate for pages that previously returned a 200 HTTP status code but currently return a 404 HTTP status code:
If Previous HTTP Status Code = 404 and If Current HTTP Status Code = 200 then Refresh every 1 day.You may want to decrease the refresh rate for pages that have returned a 404 HTTP status code for more than a week:
If Current HTTP Status Code = 404 and Count Same Consecutive HTTP Status Code > 7 then Refresh every 15 days.Defining Custom Behaviors
Configuring Markdown rules for AI Bots
To improve the efficiency in which SpeedWorkers serves pages to AI bots, identify Markdown options on the Configure AI Bots tab.
To serve Markdown to AI bots:
Select the bots to serve Markdown from the dropdown list. SpeedWorkers will serve HTML and Markdown, as shown in the preview below. If the bot explicitly requests Markdown, then SpeedWorkers will only return Markdown.
To serve Markdown to all bots requesting Markdown, regardless of the bots selected here, enable the "Markdown-only response" option.
👉 Traditional search bots will receive fully-rendered HTML from the SpeedWorkers cache when you choose to serve Markdown to AI bots.
Read next: