🛠 This document provides technical details of Botify's PageWorkers solution.
Overview
PagesWorkers is Botify's solution that allows Search Engine Optimization (SEO) teams to test and implement optimizations for search bots. SEO teams can access a Botify interface to set up optimizations and track their performance over time. Technically, modifications are inserted into the HTML of your site pages through an asynchronous JavaScript snippet provided by Botify.
How it Works
This section explains how the PageWorkers API processes the request and delivery of the page optimizations.
The Request
The following events occur when a browser loads a page that includes the Botify JS snippet:
The page renders and calls the Botify JS snippet located at: https://tags.pw.adn.cloud/{websiteId}/activation.js
Botify sends the client snippet to the page.
The snippet executes, collecting all page links and sends them in a request to the PageWorkers API through our CDN.
The Delivery
When the PageWorkers API receives the request, the following events occur:
Depending on the configuration, the snippet runs all optimizations it receives simultaneously, delivering modifications to bots only or bots and users.
The snippet adds module markers to the DOM for all elements optimized by PageWorkers to ensure the optimized elements stay on the page.
The snippet sends a log to the PageWorkers API, identifying what ran successfully and any errors encountered.
Snippet Details
Served and globally available via CloudFront
Cached for 5 mins
Size is ~40kB, gzipped
JavaScript is minified
Compatibility
The Botify JavaScript snippet is compatible with all modern browsers used by bots. PageWorkers is compatible with other scripts that dynamically edit page content, as long as you are not optimizing the same page elements.
Performance
PageWorkers is a non-blocking script because it is designed to be loaded asynchronously. Deploying the PageWorkers API in an autoscaling group allows us to scale up and down elastically to meet the workload.
The Botify JS snippet is built to have a negligible impact on the performance of any page on which it is embedded. On average, the snippet takes less than 45 ms to load. Because we load the snippet asynchronously, it does not prevent page content from being rendered, and the CDN caches and serves the JavaScript as close as possible to website visitors.
PageWorkers maintains your user experience and does not impact your pages' Core Web Vitals rating. In a benchmark study of ten sites, we demonstrated no impact on performance metrics after deploying search-driven optimizations with PageWorkers.
Data Management
The Botify JS snippet does not alter your page code. Optimizations are applied on top of your pages, which allows you to remove, deactivate, or change PageWorkers optimizations from Botify at any time, leaving your pages in their original state.
Data Exchanged
Upon execution in-browser, PageWorkers sends a secured HTTP request to its API to transmit the following information:
Page URL & embedded links: To determine the location for the optimizations.
Bot ID: Identifies the specific bot requesting the page. For user requests, this ID is blank for privacy protection.
Website ID: A unique identifier assigned by Botify and used in your Botify JS snippet URL.
Execution details: The date of the optimization execution, executed PageWorkers modules, whether the optimization was executed on desktop or mobile, and any error messages.
DOM Manipulation
The Botify JS snippet can modify any element in the page DOM using modern DOM APIs to create/update/remove elements responsibly based on the optimization parameters. We listen to the DOMContentLoaded event to determine when to start executing optimizations, and we remove the event listeners when no longer required. To limit our impact on the page as much as possible, PageWorkers does not modify the global object or any standard method in the DOM.
To counteract external changes to elements optimized by PageWorkers, the Botify JS snippet observes DOM changes and reapplies any overridden optimizations. Instead of performing a full content comparison, the snippet identifies changes by adding module markers to the DOM for the elements modified by PageWorkers. If a marked element is the target of any change while listening to the DOM, the snippet reapplies the changes and overrides all previously made changes.
Markers added to elements optimized by PageWorkers
To mitigate any performance impact of reapplying optimizations, the number of reruns is limited to five, and the event listener stops after 2.5 seconds; however, these can be adjusted by contacting Botify Support.
Botify JS snippet reapplies optimizations after an external event overrides them
PageWorkers does not write any messages in the console unless the administrator has activated the Debug mode on their terminal.
PageWorkers and Privacy
Consent-Exempt Classification: The Botify JS snippet, as part of the solution, is not considered a cookie and may fall outside the strict consent rules that typically apply to cookies. Based on customer usage, needs, and compliance requirements, it is generally categorized as an audience measurement/analytics tool and often qualifies as consent-exempt under European Data Protection Board (EDPB) guidelines.
Limited Processing of Personal Data: PageWorkers, as part of the Botify solution, may process data, primarily IP addresses of website visitors (which some jurisdictions consider personal data). This processing is carried out for the limited purpose of delivering the customer’s JS snippet to the website visitor, as required for effective delivery of the Services that the customer has subscribed to from Botify. Botify does not further store or process the IP addresses of those website visitors. The customer is encouraged to evaluate, for itself, any compliance requirements regarding the disclosure of this limited processing of personal data.
Anonymized Data: PageWorkers, within the Botify solution, primarily focuses on processing anonymized data for statistical purposes and does not aim to track users over time.
Confidentiality and Data Transfer: Botify, as part of its commitment to data privacy, maintains confidentiality for all data processed through the Botify JS snippet within the Botify solution. Specific data transfer and confidentiality requirements should be evaluated in the context of your organization and applicable regulations. All data processed is confidential and not transferred to third parties except those detailed in the Botify Data Protection Agreement (DPA).
Security Measures
The PageWorkers API is heavily protected from attacks by the following:
The API is encrypted and resilient to DDoS attacks.
PageWorkers configurations are protected in an AWS S3 bucket with strict access control (write-only access by the Botify app/API and read-only access by the PageWorkers Delivery API). Everything in the bucket is versioned.
Error Handling
In the unlikely event that Botify is experiencing an unexpected service interruption, PageWorkers is designed with the "fail securely" mode, meaning your pages will continue to be accessible, and optimizations will not be executed. If the CDN stops responding, the JavaScript snippet will not load, which will not prevent your original source page from loading. If the API stops responding, the JavaScript snippet will load, though it cannot execute the optimizations.
Contact Support
If you have questions, please contact Support.
See also: