Guides

Alation

Suggest Edits

Bigeye integrates with Alation via the Bigeye Data Health Agent.

📘

Version requirement

Required Alation API Version: greater than 2022.3

How it works

The Health tab will contain a list of Bigeye Metrics (rules) each with an associated column name (object name), status, the current observed value for the metric, description of the metric, and last updated timestamp. Users can click on the Rule link to access the metric in Bigeye, where they can get additional detail about an alert and review issues that have been raised for an alert. They can also click on the object name to access the Alation Column page and see the status for each.

Source Configuration

In order to identify the appropriate tables in Alation, users will need to add the data source ID from Alation to the corresponding source in Bigeye. To do so, a user with the Admin or Owner role must select Enable Alation Integration under Settings > Advanced Settings > Integrations in Bigeye.

Customer Credential

Each customer needs to provide the authentication detail for both Bigeye and their external catalog. Follow the below credential format for Alation.

📘

Multiple workspaces

An additional parameter called bigeye_workspace_id needs to be added to the credential below when multiple workspaces are in place. See below on where to find it. In addition, the customer will need a separate credential per workspace. Although not required, if each credential has its own unique Alation refresh token, then the integration can update the token automatically when it is close to expiration.

🚧

API Key support

When using an API key for Bigeye authentication, the parameter bigeye_workspace_id MUST be configured as part of the credential.

{
	"alation_user_id": 1234,
	"alation_refresh_token": "1234sometoken1234",
	"alation_base_url": "https://myorg.alation.com",
	"bigeye_username": "[email protected]",
	"bigeye_password": "myserviceaccountpasswordisfake",
	"bigeye_base_url": "https://app.bigeye.com",
	"bigeye_api_key": "bigeye_apk_somekey",
}

NOTE: When using an API key for Bigeye, you do not need the parameters bigeye_username or bigeye_password.

Sourcing Credential Details

  1. Basic auth credential for Bigeye.
    • Bigeye URL
    • Bigeye Service Account User ID
      • Bigeye user IDs are email addresses
    • Bigeye Service Account Password
      • ********For Bigeye SSO:********
        1. Go to https://<customer base URL>/request-password-reset?email=<email>
        2. Set password
    • Bigeye Workspace ID
  2. API credential for external catalog.
    • Alation

  • Alation Base URL
    • https://CUSTOMER.alationcatalog.com for cloud SaaS OR the customer’s vanity URL.
  • Alation User ID (Minimum role of Source Admin)
    • Find this in the URL for the User Profile
      • https://<<customer>>.alationcatalog.com/user/1234/ 1234 is the user ID
  • Alation Refresh Token
    • Create a refresh token under the Account Settings :: Authorization menu. Ensure you are creating a refresh token and not an API token.

Optional Features

Bigeye can surface additional information in Alation such as Virtual Tables, and data quality summaries at the data source and schema levels of the catalog. More info on the features below.

Bigeye Virtual Schema

Virtual tables with monitoring can appear in Alation by utilizing the catalog’s Virtual Data Source and Technical Metadata features. The integration searches for a virtual data source called Bigeye Virtual Schema. If it exists, tables and metrics will appear in the data source on the next run.

📘

Table names in Alation

How tables appear in Alation will depend on if the bigeye_workspace_id parameter is set in the credential above.

Ex:
When it is not set, tables will follow the naming convention

Bigeye.<bigeye_source_name>.<virtual_table_name>

When it is set, tables follow the form below

<bigeye_workspace_name>.<bigeye_source_name>.<virtual_table_name>

🚧

Naming conventions

Virtual table names cannot contain special characters in Alation; e.g. ., /, :, etc. It causes failures with the data health API. Keep that in mind when deciding on names for workspaces, sources, and virtual tables within Bigeye.

Create virtual data source

To create a virtual data source in Alation choose Sources from the Apps dropdown. From there, click +Add… and select Virtual Data Source

The title of the data source MUST be Bigeye Virtual Schema; i.e. Bigeye virtual schema or anything else will not work. On the next page set Database Type as Custom DB and select the check box for Use Catalog in Table Structure

After the next run of the integration, users should see virtual tables that are monitored in Bigeye in the newly created virtual data source. In addition, the SQL used to create them will also appear in Alation for users to better understand the virtual table.

Bigeye Data Quality Summary

The integration can provide summaries in Alation based on Bigeye metrics. These appear at the data source and schema levels of the catalog. The summaries utilize a rich text, custom field in Alation called Bigeye Data Quality Summary.

The data source summary tracks the number of metrics deployed and scores them along with a color coded status.

The schema summary groups metrics by dimension and scores them along with a color coded status.

Create custom field

To create the custom field in Alation navigate to settings in the top right corner. Under the Catalog Admin section choose Customize Catalog. In the Custom Fields tab, scroll down to find Rich Texts. Click the + sign to add a new field. It MUST be named Bigeye Data Quality Summary; i.e. Bigeye data quality summary or anything else will not work.

After that has been saved, click on the Custom Templates tab. Under Data Object Templates, the custom field needs to be added to the Data Source and Schema objects. Choose Insert… below Description for each and select Custom Field. Filter on Bigeye and find the Bigeye Data Quality Summary field.

Bigeye Delta Summary

The integration can provide summaries in Alation based on deltas in Bigeye. These appear at the table level of the catalog. The summaries utilize a rich text, custom field in Alation called Bigeye Delta Summary. It is a high level overview of the delta. See the steps above for creating a custom field. The field MUST be named Bigeye Delta Summary; i.e. Bigeye delta summary or anything else will not work. After the field has been created, add it to the Table object under Data Object Templates. Insert the custom field where you would like it to appear on the page.

NOTE: Currently this supports the source table of a delta only.

Updated 2 months ago