Skip to main content
All CollectionsIntegrations
Integrating with Google's Bulk Data Export
Integrating with Google's Bulk Data Export
Updated over 3 months ago

🛠 This article explains how to integrate with Google's Bulk Data Export for unsampled keywords in Botify.

Overview

To overcome Google Search Console (GSC) limitations, integrate your BigQuery data with your Botify projects. Integrating your BigQuery data provides unsampled keywords and URL performance data for Google images, videos, and news, including those that only earn a few impressions and clicks directly in RealKeywords in addition to the existing web search performance.

Benefits

Integrating with Google's bulk data export will eliminate sampling for many websites. For some large websites, some sampling may still occur, but it will be significantly decreased.

Benefits of the Bulk Data Export integration include:

  • You get the top 3.5M combinations in Botify daily (URL, keyword, device, country) for your GSC domain property vs. the top 50K with the GSC API.

  • You get the top 500K URLs in Botify daily for your GSC domain property vs. the top 5K URLs with the GSC API.*

  • These top combinations include image, video, news, Google News, and Google Discover search types.

* While the GSC API allows you to access more than 5,000 URLs, you are only guaranteed to receive the total number of clicks and impressions for the top 5,000 pages. Read more about overcoming GSC limits here.

Setting up the Bulk Data Export Integration

The Bulk Data Export integration configuration steps refer to your main GSC web property. It is important to select your top-level property for the bulk export (e.g, https://botify.com, not https://botify.com/platform). This should be your GSC domain property, which you must create in GSC if you do not already have one. Exporting your sub-properties is unnecessary since their data will be included in the main property’s export. The bulk export is limited to one property by default; please contact Support if you need to export more than one property.

👉 Since historical data is excluded from the bulk data export, we recommend you export your data as soon as possible.

Choose one of the configuration options below to set up the Bulk Data Export integration with Botify based on your preferred export location:

Prerequisites

  • If you are a new client and have not yet integrated your GSC data or are adding a new GSC property, you must complete the initial integration with GSC to activate RealKeywords and retrieve your 16 months of historical data.

  • You must be on a Botify Pro or Enterprise plan.

  • Create a GSC domain property if you do not already have one.

Botify’s BigQuery Instance

Complete the following steps to export to Botify’s BigQuery instance for your top-level GSC property:

  1. Obtain Botify’s Cloud Project Details

  2. Configure your GSC Account

  3. Configure Botify Keyword Settings

Step 1: Obtaining Botify’s Cloud Project Details

To sync your BigQuery data with your Botify project, complete the following for your GSC website property in Botify:

  1. Navigate to Settings > Keywords.
    ​

  2. Click the Configure BigQuery link next to the web property where you want to import data.
    ​

  3. In the Botify Keywords section, select the "Botify's BigQuery Instance" option to display the Cloud project ID and Dataset name. Copy these values to be used in Step 2.
    ​

  4. Click Cancel to leave the BigQuery Configuration without saving. You will return to this setup after completing Step 2.

Step 2: Configuring Your GSC Account

To initiate the bulk export:

  1. Follow the steps in the Google article's “In Search Console” section, skipping Steps 3 and 4.

  2. In Step 5, select "United States (US)" or "Europe (EU)" for the dataset location based on your proximity to the region.
    ​

Step 3: Configuring Botify Keywords Settings

48 hours after initiating the export, return to Botify Keywords settings to save the settings.

  1. In Botify, navigate to Settings > Keywords.

  2. Click the Configure BigQuery link next to the web property you selected in Step 1.

  3. Select the Botify’s BigQuery Instance option.

  4. Click Check & Save.
    ​

Your BigQuery Instance

Complete the following steps to export to your BigQuery instance for your top-level GSC property:

  1. Configure Your GCP account

  2. Configure Your GSC Account

  3. Configure Botify Keywords Settings

Step 1: Configuring Your GCP Account

  1. Select your preferred GCP project to receive the bulk export.

  2. Follow the steps in Google’s article to configure the bulk export.

  3. Then follow the step in Grant access to a dataset to assign the Big Query Data Viewer and Big Query User roles to Botify service account (501292259273-1d5th5t007a4b2ldg2nauecpjbg2q0pe@developer.gserviceaccount.com). To configure the bulk export, do this for the dataset you filled in Step 4 of the "In Search Console" section of Google’s article.

Step 2: Configuring Your GSC Account

Follow the “In Search Console” steps in Google's article to initiate the bulk export.

Step 3: Configuring Botify Keywords Settings

48 hours after initiating the export, complete the following for your top-level GSC website property:

  1. In Botify, navigate to Settings > Keywords.
    ​

  2. Click the Configure BigQuery link next to the web property where you want to import data.
    ​

  3. In the Botify Keywords section, choose Your BigQuery Instance.
    ​

  4. Add the Cloud project ID from your GCP account (from Step 1) and Dataset name from your GSC account (from Step 2).

  5. Click Check & Save.

You will start getting data from BigQuery within 24-48 hours of launching the bulk data export. Confirm the export’s completion in your GSC account.

Contact Support

If you need any assistance with setting up the Bulk Data Export integration, please contact Support.

Did this answer your question?