Connect GCP BigQuery
Create a role for Bigeye’s service account in BigQuery
- In Google Cloud Platform console, click IAM & Admin, and go to the Roles section
- Select the relevant project
- Click the Create Role button at the top of the tab. See Understanding IAM custom roles for help if needed.
- Enter a name
- Change the Role launch stage to "General Availability".
- Click Add Permissions and add the permissions specified below to the role. To make the process faster, consider filtering the permission list by the role BigQuery Admin.
bigquery.bireservations.get bigquery.capacityCommitments.get bigquery.capacityCommitments.list bigquery.config.get bigquery.datasets.create bigquery.datasets.get bigquery.datasets.getIamPolicy bigquery.jobs.create bigquery.jobs.list bigquery.jobs.listAll bigquery.models.list bigquery.readsessions.create bigquery.readsessions.getData bigquery.readsessions.update bigquery.reservationAssignments.list bigquery.reservationAssignments.search bigquery.reservations.get bigquery.reservations.list bigquery.routines.list bigquery.routines.get bigquery.savedqueries.get bigquery.savedqueries.list bigquery.tables.list bigquery.tables.get bigquery.tables.getData bigquery.transfers.get bigquerymigration.translation.translate resourcemanager.projects.get resourcemanager.projects.list
Create a service account in BigQuery
- In your Google Cloud Platform console, under IAM & Admin, go to the Service Accounts section
- Click the Create Service Account button at the top of the tab
- Give the account a name, you may want to name it “Bigeye” so you can remember it’s purpose later
- Assign the role you previously created to the service account and continue.
- Once the service account is created, create a key by clicking Mange keys, Add key, Create new key.
- Select JSON as the type and click Create. Save the JSON file downloaded.
Add BigQuery service account as a new data source in Bigeye
- In Bigeye, under Catalog, click Add source
- Select BigQuery
- Input the following information
- Name: Alias for this data source in Bigeye
- Host: BigQuery host URL
- Timeout: default timeout for Bigeye queries
- Private Key File: upload the file created from the BigQuery key
- Target BigQuery Project ID: this is an optional field, only for advanced setup. Please see below for details.
- Click Validate Source
- Test connection: This step will query your warehouse to ensure that the user you provided has the permissions necessary to read data and run all Bigeye metric types. If an error is returned, return GCP console and ensure the service account is permissioned correctly per the above instructions.
- Add metrics: If test connection is successful, you will have the option to deploy Metadata Metrics on your schemas. We recommend broad coverage of these metrics across your warehouse to detect the most common data quality issues.
- Once you’ve added your source, Bigeye will start profiling your data to recommend autometrics. It can take up to 24 hours for this process to complete. Proceed to the Getting Started for next steps.
This is only for users who want to have a service account that resides and runs queries in a different project than where the data resides. This is useful for when you want to track or limit Bigeye's BigQuery usage. Let's call the user project
project-b, and the data project
First, go to
project-b and follow the instructions above to create a service account. However, give it the following permissions. Copy the
Service account ID here for use later.
Then, go to
project-a, and follow the instructions above to create a role for Bigeye, but exclude the 2 jobs permissions above.
Then, go to IAM in the left navigation, click on Grant Access, and paste the
Service Account ID from before in the principals text field. Add the role you created for Bigeye in the previous step to this principal. Your service account is now ready.
When you add the warehouse to Bigeye, make sure to enter
project-b in the
Target BigQuery Project ID field when creating the warehouse.
Updated 4 months ago