🛠 This document provides technical details of Botify's PageWorkers solution.
Overview
PagesWorkers is Botify's solution that allows Search Engine Optimization (SEO) teams to test and implement optimizations for search bots. SEO teams can access a Botify interface to set up optimizations and track their performance over time. Technically, modifications are inserted into the HTML of your site pages through an asynchronous JavaScript tag provided by Botify.
How it Works
This section explains how the PageWorkers API processes the request and delivery of the page optimizations.
The Request
The following events occur when a browser loads a page that includes the PageWorkers tag:
The page renders and calls the PageWorkers tag located at: https://tags.pw.adn.cloud/{websiteId}/activation.js
Botify sends the client tag to the page.
The tag executes, collecting all page links and sends them in a request to the PageWorkers API through our CDN.
The Delivery
When the PageWorkers API receives the request, the following events occur:
Depending on the configuration, the tag runs all optimizations it receives simultaneously, delivering modifications to bots only or bots and users.
The tag adds module markers to the DOM for all elements optimized by PageWorkers to ensure the optimized elements stay on the page.
The tag sends a log to the PageWorkers API, identifying what ran successfully and any errors encountered.
Tag Details
Served and globally available via CloudFront
Cached for 5 mins
Size is ~40kB, gzipped
JavaScript is minified
Compatibility
The PageWorkers JavaScript tag is compatible with all modern browsers used by bots. PageWorkers is compatible with other scripts that dynamically edit page content, as long as you are not optimizing the same page elements.
Performance
PageWorkers is a non-blocking script because it is designed to be loaded asynchronously. Deploying the PageWorkers API in an autoscaling group allows us to scale up and down elastically to meet the workload.
The PageWorkers tag is built to have a negligible impact on the performance of any page on which it is embedded. On average, the tag takes less than 45 ms to load. Because we load the tag asynchronously, it doesn't prevent page content from being rendered, and the CDN caches and serves the JavaScript as close as possible to website visitors.
PageWorkers maintains your user experience and does not impact your pages' Core Web Vitals rating. In a benchmark study of ten sites, we demonstrated no impact on performance metrics after deploying search-driven optimizations with PageWorkers.
Data Management
The PageWorkers tag does not alter your page code. Optimizations are applied on top of your pages, which allows you to remove, deactivate, or change PageWorkers optimizations from Botify at any time, leaving your pages in their original state.
Data Exchanged
Upon execution in-browser, PageWorkers sends a secured HTTP request to its API to transmit the following information:
Page URL & embedded links: To determine the location for the optimizations.
Bot ID: Identifies the specific bot requesting the page. For user requests, this ID is blank for privacy protection.
Website ID: A unique identifier assigned by Botify and used in your PageWorkers tag URL.
Execution details: The date of the optimization execution, executed PageWorkers modules, whether the optimization was executed on desktop or mobile, and any error messages.
DOM Manipulation
The PageWorkers tag can modify any element in the page DOM using modern DOM APIs to create/update/remove elements responsibly based on the optimization parameters. We listen to the DOMContentLoaded event to determine when to start executing optimizations, and we remove the event listeners when no longer required. To limit our impact on the page as much as possible, PageWorkers does not modify the global object or any standard method in the DOM.
To counteract external changes to elements optimized by PageWorkers, the PageWorkers tag observes DOM changes and reapplies any overridden optimizations. Instead of performing a full content comparison, the tag identifies changes by adding module markers to the DOM for the elements modified by PageWorkers. If a marked element is the target of any change while listening to the DOM, the tag reapplies the changes and overrides all previously made changes.
Markers added to elements optimized by PageWorkers
To mitigate any performance impact of reapplying optimizations, the number of reruns is limited to five, and the event listener stops after 2.5 seconds; however, these can be adjusted by contacting Botify Support.
PageWorkers tag reapplies optimizations after an external event overrides them
PageWorkers does not write any messages in the console unless the administrator has activated the Debug mode on their terminal.
PageWorkers and Privacy
Consent-Exempt Classification: The PageWorkers JS tag, as part of the solution, is not considered a cookie and may fall outside the strict consent rules that typically apply to cookies. Based on customer usage, needs, and compliance requirements, it is generally categorized as an audience measurement/analytics tool and often qualifies as consent-exempt under European Data Protection Board (EDPB) guidelines.
Limited Processing of Personal Data: PageWorkers, as part of the Botify solution, may process data, primarily IP addresses of website visitors (which some jurisdictions consider personal data). This processing is carried out for the limited purpose of delivering the customer’s JS tag to the website visitor, as required for effective delivery of the Services that the customer has subscribed to from Botify. Botify does not further store or process the IP addresses of those website visitors. The customer is encouraged to evaluate, for itself, any compliance requirements regarding the disclosure of this limited processing of personal data.
Anonymized Data: PageWorkers, within the Botify solution, primarily focuses on processing anonymized data for statistical purposes and does not aim to track users over time.
Confidentiality and Data Transfer: Botify, as part of its commitment to data privacy, maintains confidentiality for all data processed through the PageWorkers JS tag within the Botify solution. Specific data transfer and confidentiality requirements should be evaluated in the context of your organization and applicable regulations. All data processed is confidential and not transferred to third parties except those detailed in the Botify Data Protection Agreement (DPA).
Security Measures
Botify protects the integrity of the PageWorkers tag by providing you with the hash value of the JavaScript tag. This allows you to implement control mechanisms like CSP hash or subresource integrity on your web server to enable browsers to verify that the PageWorkers tag is delivered without unexpected manipulation. Further, the whole communication is encrypted with TLS v1.2+ end-to-end when the tag is sent to the PageWorkers API.
The PageWorkers API is heavily protected from attacks by the following:
The API is encrypted and resilient to DDoS attacks.
PageWorkers configurations are protected in an AWS S3 bucket with strict access control (write-only access by the Botify app/API and read-only access by the PageWorkers Delivery API). Everything in the bucket is versioned.
Error Handling
In the unlikely event that Botify is experiencing an unexpected service interruption, PageWorkers is designed with the "fail securely" mode, meaning your pages will continue to be accessible, and optimizations will not be executed. If the CDN stops responding, the JavaScript tag will not load, which will not prevent your original source page from loading. If the API stops responding, the JavaScript tag will load, though it cannot execute the optimizations.
Contact Support
If you have questions, please contact Support.
See also: