dbt Cloud and Core

Supported Features

Platforms

dbt Cloud using automated API calls

• dbt Cloud account with API access
• dbt Cloud API token (Service Token or User Token with Job Viewer Permissions)
• Active dbt projects with jobs that generate artifacts

dbt Core using local file parsing

• Access to dbt project directory containing:
  • manifest.json
  • run_results.json
  • catalog.json

External monitors (dbt tests)

Automated synchronization of dbt test results with Bigeye's monitoring platform. This integration will create external monitors in Bigeye for each dbt test defined in your dbt project, and automatically submits test results from dbt runs to these monitors.

Lineage / Metadata extraction

The connector parses run artifacts and updates lineage and metadata for the associated dbt project in your Bigeye catalog.

Ephemeral model lineage

The connector traces lineage through ephemeral dbt models — models configured with materialized='ephemeral' that compile as CTEs rather than creating physical tables. When an ephemeral model sits between a source and a materialized model, the connector resolves the full chain so that lineage connects the original source directly to the downstream materialized output. This includes chains of multiple ephemeral models.

Configuration Parameters

Create a properties file (for example, dbt.properties) with your connection configuration. The properties file uses a numbered suffix convention (.1, .2, etc.) to support multiple connection configurations.

Sample Properties File

#################################
## SAMPLE DBT CLOUD PROPERTIES ##
#################################
environment.name.1=dbt Cloud
bigeye.host.1=https://app.bigeye.com
bigeye.apikey.1=bigeye_pak_acbdefg123456
bigeye.integration.apikey.1=bigeye_integration_abcdef123456
bigeye.allowed.workspaces.1=123
dbt.connectiontype.1=DBTConnectionType.CLOUD
dbt.dbtype.1=Snowflake
dbt.host.1=nl526.us1.dbt.com
dbt.apikey.1=dbtu_123abc567def89zxy
dbt.accounts.include.1=
dbt.accounts.exclude.1=
dbt.projects.include.1=Bigeye Internal Snowflake
dbt.projects.exclude.1=
dbt.jobs.include.1=Refresh Bigeye Internal Models,Run tests
dbt.jobs.exclude.1=
dbt.tests.default.dimension.1=Validity
[email protected],[email protected]
dbt.tests.slack.notification.list.1=#data-alerts,#example-2
dbt.tests.webhook.url.1=https://hooks.example.com/webhook/abc123
dbt.tests.webhook.headers.1=Authorization:Bearer token123,X-Custom-Header:value     

#################################
## SAMPLE DBT CORE PROPERTIES ##
#################################
environment.name.2=dbt Core
bigeye.host.2=https://app.bigeye.com
bigeye.apikey.2=bigeye_pak_acbdefg123456
bigeye.integration.apikey.2=bigeye_integration_abcdef123456
bigeye.allowed.workspaces.2=123
dbt.connectiontype.2=DBTConnectionType.CORE
dbt.local.manifests.location.2=/app/config/artifacts

Custom repository sync (dbt Core)

When using dbt Core, the connector supports syncing artifacts from a custom repository. This is useful if your CI/CD pipeline publishes dbt artifacts (manifest.json, catalog.json, run_results.json) to a shared location such as a Git repository, S3 bucket mount, or network share.

Point the dbt.local.manifests.location property to the directory containing your artifacts. If run_results.json is stored separately, use the dbt.local.runresults.location property to specify its path.

#####################################
## SAMPLE DBT CORE W/ CUSTOM REPO  ##
#####################################
environment.name.1=dbt Core - Custom Repo
bigeye.host.1=https://app.bigeye.com
bigeye.apikey.1=bigeye_pak_acbdefg123456
bigeye.integration.apikey.1=bigeye_integration_abcdef123456
bigeye.allowed.workspaces.1=123
dbt.connectiontype.1=DBTConnectionType.CORE
dbt.dbtype.1=Snowflake
dbt.local.manifests.location.1=/mnt/dbt-artifacts/prod
dbt.default.warehouse.id.1=456

Note Setting dbt.default.warehouse.id improves performance by limiting table lookups to a single warehouse instead of searching all configured warehouses.

Advanced Configuration

Per-Test Configuration with Meta Tags

You can override default monitor settings for individual tests using dbt's meta configuration. Add meta tags to your test definitions in dbt YAML files:

models:
  - name: orders
    columns:
      - name: order_id
        tests:
          - unique:
              meta:
                bigeye-dimension: "Uniqueness"
                bigeye-email-notification-list: "[email protected]"
                bigeye-slack-notification-list: "#critical-data-alerts"
          - not_null:
              meta:
                bigeye-dimension: "Completeness"
                bigeye-email-notification-list: "[email protected]"

      - name: created_at
        tests:
          - not_null:
              meta:
                bigeye-dimension: "Completeness"
                bigeye-slack-notification-list: "#data-quality"
                bigeye-webhook-url: "https://hooks.example.com/special-webhook"
                bigeye-webhook-headers: "X-Custom:value"

Supported Meta Tags

Dimension Assignment Strategy

The connector uses the following priority for assigning dimensions to monitors:

1. Test-level meta tag: bigeye-dimension in test definition (highest priority)
2. Default from properties: dbt.tests.default.dimension.{N} in properties file
3. "Custom" dimension: If no configuration and "Custom" dimension exists
4. No dimension: If none of the above are found (lowest priority)

Notification Channel Strategy

The connector uses the following priority for notification channels:

1. Test-level meta tags: bigeye-email-notification-list, bigeye-slack-notification-list , bigeye-webhook-url, and bigeye-webhook-headers(highest priority)
2. Default from properties: dbt.tests.email.notification.list.{N}, dbt.tests.slack.notification.list.{N}, dbt.tests.webhook.url.{N}, and dbt.tests.webhook.headers.{N}
3. No notifications: If neither are configured (lowest priority)

Warehouse and Dataset Assignment

The connector automatically determines the correct Bigeye warehouse and dataset (table) for each test by:

1. Resolving test dependencies: Follows depends_on relationships to find the actual model/source being tested
2. Querying Bigeye catalog: Looks up the corresponding table using database.schema.table identifiers
3. Using default warehouse (optional): If dbt.default.warehouse.id.{N} is configured, only tables in that warehouse are considered

Important: Tables must exist in Bigeye before external monitors can be created for tests on those tables. Ensure your dbt target tables are configured as data sources in Bigeye.

Running the connector

Prerequisites

Before deploying the dbt connector, ensure you have:

Infrastructure Requirements

• Minimum HW size: 1 CPU, 2GB mem.
  • AWS t3.small
  • GCP e2-small
  • Azure B1ms
• Network access for agent subnets
  • API access to Bigeye
  • Bigeye API (app.bigeye.com or your Bigeye instance URL)
  • dbt Cloud API (if using dbt Cloud)

Bigeye Requirements

• Active Bigeye workspace(s) with admin access
• Bigeye Personal API key with appropriate permissions
• Integration API key (distinct from standard API key - see Creating an Integration API Key)(If using Agent CLI, this will be created automatically)
• Data sources configured in Bigeye that correspond to your dbt target database(s)
• Tables from your dbt project already cataloged in Bigeye

Running the connector with the Agent CLI (recommended)

If you are using the Bigeye Agent CLI to run the dbt connector, then you will want to ensure the following steps are taken:

1. Download the latest version of the Agent CLI using the commands detailed here.
2. Ensure both the Lineage Plus and External Monitors agents have been installed and configured using the ./bigeye-agent install command.
3. Add the dbt connection info using the ./bigeye-agent add-connector -c dbt command. This will provide a series of prompts and generated the desired dbt.properties file.
4. Run the dbt connector using the following commands:

# Run the lineage metadata features
./bigeye-agent lineage run -c dbt

# Run the external monitors features
./bigeye-agent external-monitors run -c dbt

Running the connector with Docker

If you are not using the Agent CLI, you can run the connector directly with Docker. See Agent-based Connections for Docker installation instructions.

1. Log in to Docker Hub and pull the connector image:

   Bash

   docker login --username bigeyedata --password <docker access token>
   docker pull docker.io/bigeyedata/source-connector:latest

2. Run the external monitors integration:

   Bash

   docker run --rm -v /path/to/dbt_config:/app/config \
     --entrypoint bash bigeyedata/source-connector:latest \
     -c "bigeye-connector monitor -c dbt -p /app/config/dbt.properties"

3. Run the lineage / metadata integration:

   Bash

   docker run --rm -v /path/to/dbt_config:/app/config \
     --entrypoint bash bigeyedata/source-connector:latest \
     -c "bigeye-connector run -c dbt -p /app/config/dbt.properties"

Creating an Integration API Key

The External Monitors feature requires a special Integration API key, which is different from the standard Bigeye API key. This key has specific permissions for creating and managing external monitors.

API Request

Use the following API request to create an Integration API key:

curl --location 'https://app.bigeye.com/api/v1/agent-api-keys' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: apikey YOUR_PERSONAL_API_KEY' \
  --data '{
    "name": "External Monitors Integration Key",
    "description": "API key for external monitors integration",
    "type": "AGENT_API_KEY_TYPE_INTEGRATION"
  }'