Connect GCP BigQuery

Suggest Edits

Create a role for Bigeye’s service account in BigQuery

  1. In Google Cloud Platform console, click IAM & Admin, and go to the Roles section
  2. Select the relevant project
  3. Click the Create Role button at the top of the tab. See Understanding IAM custom roles for help if needed.
    1. Enter a name
    2. Change the Role launch stage to "General Availability".
    3. Click Add Permissions and add the permissions specified below to the role. To make the process faster, consider filtering the permission list by the role BigQuery Admin.

Permissions:

bigquery.bireservations.get
bigquery.capacityCommitments.get
bigquery.capacityCommitments.list
bigquery.config.get
bigquery.datasets.create
bigquery.datasets.get
bigquery.datasets.getIamPolicy     
bigquery.jobs.create
bigquery.jobs.list
bigquery.jobs.listAll
bigquery.models.list
bigquery.readsessions.create
bigquery.readsessions.getData
bigquery.readsessions.update
bigquery.reservationAssignments.list
bigquery.reservationAssignments.search
bigquery.reservations.get
bigquery.reservations.list
bigquery.routines.list
bigquery.routines.get
bigquery.savedqueries.get
bigquery.savedqueries.list
bigquery.tables.list
bigquery.tables.get
bigquery.tables.getData
bigquery.transfers.get
bigquerymigration.translation.translate
resourcemanager.projects.get
resourcemanager.projects.list

Create a service account in BigQuery

  1. In your Google Cloud Platform console, under IAM & Admin, go to the Service Accounts section
  2. Click the Create Service Account button at the top of the tab
  3. Give the account a name, you may want to name it “Bigeye” so you can remember it’s purpose later
  4. Assign the role you previously created to the service account and continue.
  5. Once the service account is created, create a key by clicking Mange keys, Add key, Create new key.
    1. Select JSON as the type and click Create. Save the JSON file downloaded.

Add BigQuery service account as a new data source in Bigeye

  1. In Bigeye, under Catalog, click Add source
  2. Select BigQuery
  3. Input the following information
    1. Name: Alias for this data source in Bigeye
    2. Host: BigQuery host URL
    3. Timeout: default timeout for Bigeye queries
    4. Private Key File: upload the file created from the BigQuery key
    5. Target BigQuery Project ID: this is an optional field, only for advanced setup. Please see below for details.
  4. Click Validate Source
  5. Test connection: This step will query your warehouse to ensure that the user you provided has the permissions necessary to read data and run all Bigeye metric types. If an error is returned, return GCP console and ensure the service account is permissioned correctly per the above instructions.
  6. Add metrics: If test connection is successful, you will have the option to deploy Metadata Metrics on your schemas. We recommend broad coverage of these metrics across your warehouse to detect the most common data quality issues.
  7. Once you’ve added your source, Bigeye will start profiling your data to recommend autometrics. It can take up to 24 hours for this process to complete. Proceed to the Getting Started for next steps.

Advanced setup

This is only for users who want to have a service account that resides and runs queries in a different project than where the data resides. This is useful for when you want to track or limit Bigeye's BigQuery usage. Let's call the user project project-b, and the data project project-a.

First, go to project-b and follow the instructions above to create a service account. However, give it the following permissions. Copy the Service account ID here for use later.

bigquery.jobs.create
bigquery.jobs.list

Then, go to project-a, and follow the instructions above to create a role for Bigeye, but exclude the 2 jobs permissions above.

Then, go to IAM in the left navigation, click on Grant Access, and paste the Service Account ID from before in the principals text field. Add the role you created for Bigeye in the previous step to this principal. Your service account is now ready.

When you add the warehouse to Bigeye, make sure to enter project-b in the Target BigQuery Project ID field when creating the warehouse.

Updated 4 months ago