Connect GCP BigQuery

Create a role for Bigeye’s service account in BigQuery

  1. In Google Cloud Platform console, click IAM & Admin, and go to the Roles section
  2. Select the relevant project
  3. Click the Create Role button at the top of the tab. See Understanding IAM custom roles for help if needed.
    1. Enter a name
    2. Change the Role launch stage to "General Availability".
    3. Click Add Permissions and add the permissions specified below to the role. To make the process faster, consider filtering the permission list by the role BigQuery Admin.

Permissions:

bigquery.datasets.get
bigquery.datasets.getIamPolicy
bigquery.jobs.get
bigquery.jobs.list
bigquery.jobs.listAll
bigquery.jobs.create
bigquery.tables.get
bigquery.tables.getData
bigquery.tables.list
resourcemanager.projects.get

Create a service account in BigQuery

  1. In your Google Cloud Platform console, under IAM & Admin, go to the Service Accounts section
  2. Click the Create Service Account button at the top of the tab
  3. Give the account a name, you may want to name it “Bigeye” so you can remember it’s purpose later
  4. Assign the role you previously created to the service account and continue.
  5. Once the service account is created, create a key by clicking Mange keys, Add key, Create new key.
    1. Select JSON as the type and click Create. Save the JSON file downloaded.

Add BigQuery service account as a new data source in Bigeye

  1. In Bigeye, under Catalog, click Add source
  2. Select BigQuery
  3. Input the following information
    1. Name: Alias for this data source in Bigeye
    2. Host: BigQuery host URL
    3. Timeout: default timeout for Bigeye queries
    4. Private Key File: upload the file created from the BigQuery key
  4. Click Validate Source
  5. Test connection: This step will query your warehouse to ensure that the user you provided has the permissions necessary to read data and run all Bigeye metric types. If an error is returned, return GCP console and ensure the service account is permissioned correctly per the above instructions.
  6. Add metrics: If test connection is successful, you will have the option to deploy Metadata Metrics on your schemas. We recommend broad coverage of these metrics across your warehouse to detect the most common data quality issues.
  7. Once you’ve added your source, Bigeye will start profiling your data to recommend autometrics. It can take up to 24 hours for this process to complete. Proceed to the Getting Started for next steps.