📘 This article provides information on how to set the cache behaviors to optimize your cache refresh in SpeedWorkers. SpeedWorkers is part of Botify's Activation Suite, available as an option with a Botify Pro or Enterprise plan.
Overview
The Cache Behavior settings in SpeedWorkers determine how the pages in your scope are fetched and rendered and how long pages remain in page inventory. By setting different cache rates and rules for different types of content, you can optimize your page inventory by allowing SpeedWorkers to render more efficiently. You can set both default cache behaviors and custom behaviors that override the default behaviors for specific pages.
Access the Cache Behavior settings by navigating to Activation > Speedworkers > Settings > Cache Behaviors:
Refresh Capacity Rate
A banner displays the current refresh capacity rate at the top of the Cache Behaviors settings page. This provides visibility into whether SpeedWorkers can refresh all pages in your inventory based on current settings. Please refer to the Managing Refresh Capacity in SpeedWorkers article for more information.
Defining the Default Behavior
The default behavior defines the cache refresh behavior for all URLs in your scope that do not match a specified custom behavior. The default behavior includes the following rules:
To define the default behavior:
If settings are not in Edit mode, refer to the Defining SpeedWorkers Settings article to enable editing.
Click the pencil icon in the Default behavior block:
Define the default rules, and then click Next.
Optionally, add advanced rules, and then click Save.
Defining Default Rules
The default rules define the rate and priority at which pages are refreshed, whether separate versions for mobile and desktop are cached, and how JavaScript redirections should be handled.
A JS redirection is when the JavaScript code changes the URL of the current page without making the browser reload the page. When SpeedWorkers detects a JS redirection, you can have SpeedWorkers not cache the page or cache it as a 301 or 302 redirection.
To define default rules:
Select the number and time increment to identify the rate at which pages in the SpeedWorkers inventory should be refreshed.
If an issue prevents page refresh at the above rate, the pages will enter a "stale" state. Select the number and time increment to identify how long after the latest refresh attempt to serve the stale pages. We recommend setting this at approximately 25% of the refresh rate (e.g., for a weekly refresh, serve stale pages for 2 days). This allows SpeedWorkers to continue serving the pages to bots instead of the pages being moved directly into an expired state. Learn more about page states in the Understanding the Page Lifecycle in SpeedWorkers article.
Click the Open Advanced Settings link to identify the action to occur when a JavaScript redirection is encountered on pages within the defined scope.
We recommend leaving the default "text/html" value in the Content-Type field. To change this value, click the pencil icon to enable editing and then identify the page content type.
From the JS Redirection dropdown list, select the action to occur when a JavaScript redirection is detected.
Select the priority at which these pages should be refreshed to instruct SpeedWorkers which pages to refresh first (i.e., high refresh priority pages will be refreshed first).
In the "Mobile & Desktop Versions" section:
Select the "Responsive Version" option to cache only one version of the page; this is the recommended option for sites that have responsive pages.
Select the "Desktop version/Mobile version" option to cache separate desktop and mobile page versions.
From the Mobile & Desktop rendering dropdown lists, choose whether SpeedWorkers should render pages with the default JavaScript configuration, HTML configuration, or not. The render option is the same for mobile and desktop pages when "Responsive Version" is selected and separate when "Desktop version/Mobile version" is selected.
To have SpeedWorkers serve clean URLs without parameters, identify the parameters to be ignored when SpeedWorkers caches pages. Type parameters in the text box, separating multiple parameters with commas, and then click the Add parameter link. Parameters are listed below the text box. Click the X next to a parameter to delete it from the list, or click the Clear all button at the right to remove all parameters.
If you identified parameters to be ignored, select an action from the dropdown list to identify the behavior for pages with removed parameters. Select the "Remove ignored parameters . . . " checkbox to remove the parameters identified above from all links, canonicals, and hreflang attributes on all pages within the scope.
Click Next to add advanced rules.
Defining Advanced Rules
Advanced Rules help you avoid refresh waste by allowing you to make rules based on HTTP code, page property, or input source. Adding these additional rules allows you to determine the behavior when a page is refreshed but the status code or page property has changed. You can create additional rules based on the following:
Current or previous HTTP status code.
Count Same Consecutive HTTP Status Code: The number of times the page returned the same status code (consecutively).
Duration Same Consecutive HTTP Status Code: How long the page has been returning the same status code, in minutes.
Current or previous page property: A page property describes the page's content or an element (e.g., a profile page or product availability). Botify can extract any value from the HTML and set it as a page property. If you are unaware of the page properties set for your project, please contact your SEO Success Manager.
Inventory input: The source from where the pages were added to the SpeedWorkers inventory.
To add an advanced rule:
Select a criterion, operator, and value to define the rule (e.g., Current HTTP Status Code equals 200).
Click the Add a condition link to add additional criteria to the rule.
Select a refresh rate, the time the page can be stale, and the refresh priority (i.e., the order in which SpeedWorkers will cache the pages).
Click the Do not update cache checkbox SpeedWorkers if you do not want the cache to update when SpeedWorkers encounters pages that fit this rule.
Optionally, click the Add an advanced rule link to add additional rules.
Click Save.
Advanced Rule Examples
You may want to increase the refresh rate for pages that previously returned a 200 HTTP status code but currently return a 404 HTTP status code:
If Previous HTTP Status Code = 404 and If Current HTTP Status Code = 200 then Refresh every 1 day.You may want to decrease the refresh rate for pages that have returned a 404 HTTP status code for more than a week:
If Current HTTP Status Code = 404 and Count Same Consecutive HTTP Status Code > 7 then Refresh every 15 days.
Defining Custom Behaviors
The Custom Cache Behavior settings in SpeedWorkers override the default cache behavior settings for the URLs within the scope that you define. Defining a custom cache behavior includes the following steps:
To define a custom behavior:
If settings are not in Edit mode, refer to the Defining SpeedWorkers Settings article to enable editing.
Click the Create custom behavior link in the Custom behavior block, or hover over an existing rule and click the pencil icon to edit.
Define the scope, and then click Next.
Set the custom behavior rules, following Steps 1 - 7 in the Defining Default Rules section, and then click Next.
Optionally, add advanced rules, and then click Save.
👉 The settings for rules in custom behaviors are inherited from the default behavior rules. Click the pencil icon next to the field to change the value.
Defining the Scope
The scope determines which pages will be impacted by the custom behavior.
To define the custom behavior scope:
Provide a title and, optionally, a description to identify the custom behavior.
From the first dropdown list, select the element by which you want to define the scope (e.g., URL).
Select the operator from the second dropdown list that determines the relationship between the element you selected above and the string to be matched (e.g., contains).
In the text field, identify the string to be matched (e.g., "/product/").
Optionally, click the Add a Condition link to add another condition to be matched for this custom behavior.
Reordering Custom Behaviors
The order in which Custom Behaviors appear in the list of rules is important. If a URL matches more than one behavior, the first matching behavior in the list will be applied, and any subsequent matching behaviors will not be executed. For this reason, you may need to change the order in which the behaviors are displayed.
To reorder custom behaviors:
In the Custom Behaviors section of the Cache Behaviors settings, click the up arrow in the behavior block to move the behavior one step higher in the list or click the down arrow to move the rule one step lower in the list.
Repeat until the behavior is at the desired location in the list.
Read next: