📘 Enrichment split testing measures how AgenticCatalog enrichment affects product performance in AI shopping destinations such as OpenAI and Google Merchant Center. It is available for each AgenticCatalog destination. AgenticCatalog is part of Botify's Activation Suite, available as an option with all Botify plans.
Overview
Enrichment split testing compares products sent with enriched values against similar products sent with their original values. For each destination, AgenticCatalog divides your enrichment-eligible products into two groups:
Variant group: Products sent to the destination with enriched values.
Control group: Products sent to the destination with the original values from your source feed.
Both groups are measured over the same period, so seasonality, platform changes, and other external factors affect them equally. The difference in performance between the groups shows the impact of enrichment itself.
👀 In this article, "variant" refers to the test group. It does not refer to product variants such as sizes or colors.
How products are assigned to groups
AgenticCatalog assigns products to the control and variant groups automatically. You set the size of the control group; you do not select individual products.
Only enrichment-eligible products are included. If your enrichment settings limit enrichment to specific categories or price ranges, the split applies only to those products. All other products belong to neither group and are sent with their original values.
The groups are balanced. AgenticCatalog distributes products so that both contain the same kind of product.
Assignments are stable. Each product is assigned based on its product ID and stays in the same group across every feed version and every push for the duration of the test.
Example
Your feed contains 5,500 products. Your enrichment settings limit enrichment to the Boots category, which contains 500 products, and you set the control group to 20%.
Products | Count | Group | Values sent to the destination |
Boots | 400 | Variant | Enriched |
Boots | 100 | Control | Original |
All other categories | 5,000 | None | Original |
Setting up a split test
You configure enrichment split testing separately in each destination's settings. Each destination can have one active split test at a time.
Navigate to the destination settings for your feed source.
In the "Control group" field, enter the percentage of enrichment-eligible products to send with original values. We recommend a 10% minimum and 50% maximum.
Click Test to check the statistical validity. The results display in the “Balance report” section. If any test fails, either:
Click Re-shuffle to redistribute the products between the control and variant groups; or
Increase the control group percentage.
Save the destination settings. The split applies to the next feed generated for that destination.
In the destination feed preview, products in the control group are tagged Control group below the product ID and show their original values. Products in the variant group show enriched values.
👉 Recommendation: Start with a 20% control group and run the test for at least two - four weeks.
Changing a split test in progress
Changing how products are assigned during a test invalidates the comparison with results collected so far. Avoid these changes until the test is complete.
Action | Effect on group assignments | Effect on results |
Save destination settings without changing the control group percentage | None. Products stay in their current groups. | None |
Change the control group percentage | Products are reassigned. A warning appears, and you can start a new test or keep the current settings. | Results before the change are not comparable with results after it. |
Click Re-shuffle | All products are reassigned to new groups. | A new comparison starts. |
Swap the control and variant groups | Control products become variant products, and variant products become control products. | A new comparison starts. Swapping can confirm a result by reversing the groups. |
How groups change when your feed changes
Existing products keep their group assignment when your source feed changes. Because products are not rebalanced, the actual share of products in the control group can drift from the percentage you configured.
New products: When a new product appears in your feed and is eligible for enrichment, AgenticCatalog assigns it to a group automatically.
Removed products: When a product leaves your feed, it leaves its group. Other products do not move to replace it. For example, if you configured a 20% control group and your feed drops many control products, the control group falls below 20%.
Changed product IDs: Group assignment is based on the product ID. A product whose ID changes is treated as a new product and can be assigned to the other group. Keep product IDs stable during a test.
Reading the results
The "Enrichment split test results" section in AgenticCatalog Insights compares the two groups for each destination.
Before the test starts, the chart shows one line for all products.
After the test starts, the chart shows two lines: the variant group as a colored line and the control group as a solid gray line. The test start date is marked on the chart.
Values are averages per product, so the groups can be compared even though the control group is smaller than the variant group.
Metrics include clicks and impressions from the destination.
If the variant group consistently averages higher than the control group, enrichment is improving product performance in that destination.
📊 Performance data from OpenAI and Google Merchant Center arrives with a delay of 24–48 hours. The most recent days in the chart can be incomplete.
Best practices
Run each test for at least two weeks. Catalogs with fewer products or lower traffic can need more than four weeks to show a clear difference.
Avoid starting a test during atypical periods, such as major sales events, which can distort results.
Do not change the control group percentage, re-shuffle, or swap groups until the test is complete.
Keep product IDs stable for the duration of the test.
Limitations
Each destination supports one active split test at a time.
You cannot test individual enrichment types, such as Q&A only. The variant group receives all configured enrichment.
Results are not tested for statistical significance.
Outlier products are not excluded from the averages, meaning a few unusually high-performing products can affect the results for their group.
Since there is no check of average traffic for both groups before products are split between the groups, we cannot determine whether one group was already performing better or worse than the other before the test.
FAQs
Can I choose which products are in the control group?
Can I choose which products are in the control group?
No. AgenticCatalog assigns products automatically to keep the groups comparable. You control only the size of the control group and, through your enrichment settings, which products are eligible.
Are control group products enriched?
Are control group products enriched?
Yes, but the enrichment is not used when products are in the control group. AgenticCatalog enriches all eligible products. The split determines only which values are sent to the destination: original values for the control group and enriched values for the variant group.
Does each destination use the same groups?
Does each destination use the same groups?
No. Each destination has its own split test, configured in its own destination settings.
Why did the share of products in my control group change?
Why did the share of products in my control group change?
Products keep their group when your feed changes. If your feed removes more products from one group than the other, the actual percentage drifts from the configured percentage. See How groups change when your feed changes.
Read next: