🛠 This article explains how to integrate with Google's Bulk Data Export for unsampled keywords in Botify.
Overview
To overcome Google Search Console (GSC) limitations, integrate your BigQuery data with your Botify projects. Integrating your BigQuery data provides unsampled keywords and URL performance data for Google images, videos, and news, including those that only earn a few impressions and clicks directly in RealKeywords in addition to the existing web search performance.
Benefits
Integrating with Google's bulk data export will eliminate sampling for many websites. For some large websites, some sampling may still occur, but it will be significantly decreased.
Benefits of the Bulk Data Export integration include:
You get the top 3.5M combinations in Botify daily (URL, keyword, device, country) for your GSC domain property vs. the top 50K with the GSC API.
You get the top 500K URLs in Botify daily for your GSC domain property vs. the top 5K URLs with the GSC API.*
These top combinations include image, video, news, Google News, and Google Discover search types.
* While the GSC API allows you to access more than 5,000 URLs, you are only guaranteed to receive the total number of clicks and impressions for the top 5,000 pages. Read more about overcoming GSC limits here.
Setting up the Bulk Data Export Integration
The Bulk Data Export integration configuration steps refer to your main GSC web property. It is important to select your top-level property for the bulk export (e.g, https://botify.com, not https://botify.com/platform). This should be your GSC domain property, which you must create in GSC if you do not already have one. Exporting your sub-properties is unnecessary since their data will be included in the main property’s export. The bulk export is limited to one property by default; please contact Support if you need to export more than one property.
👉 Since historical data is excluded from the bulk data export, we recommend you export your data as soon as possible.
Choose one of the configuration options below to set up the Bulk Data Export integration with Botify based on your preferred export location:
Botify’s BigQuery Instance: Export your BigQuery data to Botify’s GCP account.
Your BigQuery Instance: Export your BigQuery data to your Google Cloud Platform (GCP) account.
Prerequisites
If you are a new client and have not yet integrated your GSC data or are adding a new GSC property, you must complete the initial integration with GSC to activate RealKeywords and retrieve your 16 months of historical data.
You must be on a Botify Pro or Enterprise plan.
Create a GSC domain property if you do not already have one.
Botify’s BigQuery Instance
👉 With this option, Botify absorbs the GSC storage fees, but we reserve the right to delete older data.
Complete the following steps to export to Botify’s BigQuery instance for your top-level GSC property:
Step 1: Obtaining Botify’s Cloud Project Details
To sync your BigQuery data with your Botify project, complete the following for your GSC website property in Botify:
Navigate to Settings > Keywords.
Click the Configure BigQuery link next to the web property where you want to import data.
In the Botify Keywords section, select the "Botify's BigQuery Instance" option to display the Cloud project ID and Dataset name. Copy these values to be used in Step 2.
Click Cancel to leave the BigQuery Configuration without saving. You will return to this setup after completing Step 2.
Step 2: Configuring Your GSC Account
To initiate the bulk export:
Follow the steps in the Google article's “In Search Console” section, skipping Steps 3 and 4.
In Step 5, select "United States (US)" or "Europe (EU)" for the dataset location based on your proximity to the region.
Step 3: Configuring Botify Keywords Settings
48 hours after initiating the export, return to Botify Keywords settings to save the settings.
In Botify, navigate to Settings > Keywords.
Click the Configure BigQuery link next to the web property you selected in Step 1.
Select the Botify’s BigQuery Instance option.
Click Check & Save.
Your BigQuery Instance
👉 With this option, you are responsible for the GSC storage fees. You have full control over your data, including the ability to query it and integrate with other tools.
Complete the following steps to export to your BigQuery instance for your top-level GSC property:
Step 1: Configuring Your GCP Account
In this step, you configure your GCP account and grant Botify access to your dataset.
👉 The dataset you share with Botify is created in the bulk export process. This dataset always starts with searchconsole and must contain both "by URL" and "by website" tables.
Select your preferred GCP project to receive the bulk export.
Follow the steps in Google’s article to configure the bulk export.
Then follow the step in Grant access to a dataset to assign the Big Query Data Viewer and Big Query User roles to Botify service account (501292259273-1d5th5t007a4b2ldg2nauecpjbg2q0pe@developer.gserviceaccount.com). To configure the bulk export, do this for the dataset you filled in Step 4 of the "In Search Console" section of Google’s article.
Step 2: Configuring Your GSC Account
Follow the “In Search Console” steps in Google's article to initiate the bulk export.
Step 3: Configuring Botify Keywords Settings
48 hours after initiating the export, complete the following for your top-level GSC website property:
In Botify, navigate to Settings > Keywords.
Click the Configure BigQuery link next to the web property where you want to import data.
In the Botify Keywords section, choose Your BigQuery Instance.
Click Check & Save.
You will start getting data from BigQuery within 24-48 hours of launching the bulk data export. Confirm the export’s completion in your GSC account.
Contact Support
If you need any assistance with setting up the Bulk Data Export integration, please contact Support.