dbt Connector

Overview

The dbt connector collects run artifacts from your dbt projects and parses those run artifacts for lineage, metadata, and dbt tests to integrate into Bigeye. We support both dbt Core and dbt Cloud through a configurable properties file.

Supported Features

Platforms

dbt Cloud using automated API calls

• dbt Cloud account with API access
• dbt Cloud API token (Service Token or User Token with Job Viewer Permissions)
• Active dbt projects with jobs that generate artifacts

dbt Core using local file parsing

• Access to dbt project directory containing:
  • manifest.json
  • run_results.json
  • catalog.json

External monitors (dbt tests)

Automated synchronization of dbt test results with Bigeye's monitoring platform. This integration will create external monitors in Bigeye for each dbt test defined in your dbt project, and automatically submits test results from dbt runs to these monitors.

Lineage / Metadata extraction

The connector will parse the run artifacts and update lineage / metadata of the associated dbt project in your Bigeye catalog.

Configuration Parameters

Create a properties file (e.g., dbt.properties) with your connection configuration. The properties file uses a numbered suffix convention (.1, .2, etc.) to support multiple connection configurations.

Here is the complete list of all supported properties with descriptions:

Sample Properties File

#################################
## SAMPLE DBT CLOUD PROPERTIES ##
#################################
environment.name.1=dbt Cloud
bigeye.host.1=https://app.bigeye.com
bigeye.apikey.1=bigeye_pak_acbdefg123456
bigeye.integration.apikey.1=bigeye_integration_abcdef123456
bigeye.allowed.workspaces.1=123
dbt.connectiontype.1=DBTConnectionType.CLOUD
dbt.dbtype.1=Snowflake
dbt.host.1=nl526.us1.dbt.com
dbt.apikey.1=dbtu_123abc567def89zxy
dbt.accounts.include.1=
dbt.accounts.exclude.1=
dbt.projects.include.1=Bigeye Internal Snowflake
dbt.projects.exclude.1=
dbt.jobs.include.1=Refresh Bigeye Internal Models,Run tests
dbt.jobs.exclude.1=
dbt.tests.default.dimension.1=Validity
[email protected],[email protected]
dbt.tests.slack.notification.list.1=#data-alerts,#example-2
dbt.tests.webhook.url.1=https://hooks.example.com/webhook/abc123
dbt.tests.webhook.headers.1=Authorization:Bearer token123,X-Custom-Header:value     

#################################
## SAMPLE DBT CORE PROPERTIES ##
#################################
environment.name.2=dbt Core
bigeye.host.2=https://app.bigeye.com
bigeye.apikey.2=bigeye_pak_acbdefg123456
bigeye.integration.apikey.2=bigeye_integration_abcdef123456
bigeye.allowed.workspaces.2=123
dbt.connectiontype.2=DBTConnectionType.CORE
dbt.local.manifests.location.2=/app/config/artifacts

Advanced Configuration

Per-Test Configuration with Meta Tags

You can override default monitor settings for individual tests using dbt's meta configuration. Add meta tags to your test definitions in dbt YAML files:

models:
  - name: orders
    columns:
      - name: order_id
        tests:
          - unique:
              meta:
                bigeye-dimension: "Uniqueness"
                bigeye-email-notification-list: "[email protected]"
                bigeye-slack-notification-list: "#critical-data-alerts"
          - not_null:
              meta:
                bigeye-dimension: "Completeness"
                bigeye-email-notification-list: "[email protected]"

      - name: created_at
        tests:
          - not_null:
              meta:
                bigeye-dimension: "Completeness"
                bigeye-slack-notification-list: "#data-quality"
                bigeye-webhook-url: "https://hooks.example.com/special-webhook"
 								bigeye-webhook-headers: "X-Custom:value"

Supported Meta Tags

Dimension Assignment Strategy

The connector uses the following priority for assigning dimensions to monitors:

1. Test-level meta tag: bigeye-dimension in test definition (highest priority)
2. Default from properties: dbt.tests.default.dimension.{N} in properties file
3. "Custom" dimension: If no configuration and "Custom" dimension exists
4. No dimension: If none of the above are found (lowest priority)

Notification Channel Strategy

The connector uses the following priority for notification channels:

1. Test-level meta tags: bigeye-email-notification-list, bigeye-slack-notification-list , bigeye-webhook-url, and bigeye-webhook-headers(highest priority)
2. Default from properties: dbt.tests.email.notification.list.{N}, dbt.tests.slack.notification.list.{N}, dbt.tests.webhook.url.{N}, and dbt.tests.webhook.headers.{N}
3. No notifications: If neither are configured (lowest priority)

Warehouse and Dataset Assignment

The connector automatically determines the correct Bigeye warehouse and dataset (table) for each test by:

1. Resolving test dependencies: Follows depends_on relationships to find the actual model/source being tested
2. Querying Bigeye catalog: Looks up the corresponding table using database.schema.table identifiers
3. Using default warehouse (optional): If dbt.default.warehouse.id.{N} is configured, only tables in that warehouse are considered

Important: Tables must exist in Bigeye before external monitors can be created for tests on those tables. Ensure your dbt target tables are configured as data sources in Bigeye.

Running the connector

Prerequisites

Before deploying the dbt connector, ensure you have:

Infrastructure Requirements

• Minimum HW size: 1 CPU, 2GB mem.
  • AWS t3.small
  • GCP e2-small
  • Azure B1ms
• Network access for agent subnets
  • API access to Bigeye
  • Bigeye API (app.bigeye.com or your Bigeye instance URL)
  • dbt Cloud API (if using dbt Cloud)

Bigeye Requirements

• Active Bigeye workspace(s) with admin access
• Bigeye Personal API key with appropriate permissions
• Integration API key (distinct from standard API key - see Creating an Integration API Key)(If using Agent CLI, this will be created automatically)
• Data sources configured in Bigeye that correspond to your dbt target database(s)
• Tables from your dbt project already cataloged in Bigeye

Running the connector with the Agent CLI (Recommended)

If you are using the Bigeye Agent CLI to run the dbt connector, then you will want to ensure the following steps are taken:

1. Download the latest version of the Agent CLI using the commands detailed here.
2. Ensure both the Lineage Plus and External Monitors agents have been installed and configured using the ./bigeye-agent install command.
3. Add the dbt connection info using the ./bigeye-agent add-connector -c dbt command. This will provide a series of prompts and generated the desired dbt.properties file.
4. Run the dbt connector using the following commands:

# Run the lineage metadata features
./bigeye-agent lineage run -c dbt

# Run the external monitors features
./bigeye-agent external-monitors run -c dbt

Running the connector manually

Installation

Host with docker

1. Install docker if it is not already installed [CentOS][RHEL][Debian][Ubuntu]
   1. NOTE for RHEL, the official docker instructions don't quite work. Using docker's CENTOS yum repository instead is a good workaround.
   2. bash

      sudo yum install -y yum-utils
      sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
      sudo yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
      sudo systemctl start docker
      # test it 
      sudo docker run hello-world

2. You can test if docker is running by using docker info and getting a non-error response
3. Login to Docker Hub using access token provided by Bigeye and pull the bigeyedata/source-connector image:

   1. docker logout
      docker login --username bigeyedata --password <docker access token>
      docker pull docker.io/bigeyedata/source-connector:latest

Execution

To run the dbt connector, here are the steps:

1. Create a directory called dbt_config and create a properties file with the details of your connector (see details on dbt below):

   mkdir dbt_config
   cd dbt_config
   touch dbt.properties

2. Run the external monitors integration with the following command:

   docker run --rm -v /path/to/dbt_config:/app/config --entrypoint bash bigeyedata/source-connector:latest -c "bigeye-connector monitor -c dbt -p /app/config/dbt.properties"

3. Run the lineage / metadata integration with the following command:

   docker run --rm -v /path/to/dbt_config:/app/config --entrypoint bash bigeyedata/source-connector:latest -c "bigeye-connector run -c dbt -p /app/config/dbt.properties"

   
   

Creating an Integration API Key

The External Monitors feature requires a special Integration API key, which is different from the standard Bigeye API key. This key has specific permissions for creating and managing external monitors.

API Request

Use the following API request to create an Integration API key:

curl --location 'https://app.bigeye.com/api/v1/agent-api-keys' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: apikey YOUR_PERSONAL_API_KEY' \
  --data '{
    "name": "External Monitors Integration Key",
    "description": "API key for external monitors integration",
    "type": "AGENT_API_KEY_TYPE_INTEGRATION"
  }'