Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.

[0.14.2] - 2026-08-19

Security

Bumped cryptography to >=50.0.0, fixing CVE-2026-69247.

[0.14.1] - 2026-08-19

Security

Bumped GitPython to >=3.1.58, fixing a family of argument-injection advisories (GHSA-9rj7-rf2p-w77r, GHSA-539m-9xh6-q6rr, GHSA-p538-c434-8v24, GHSA-3f7w-8rr8-f37f, GHSA-hh9p-6wh2-4mfc, GHSA-4gmw-gg2m-w46p, GHSA-wvpp-8hx9-p66j, GHSA-hmq2-w58f-27jc, GHSA-jm78-9fvv-mhgr).

[0.14.0] - 2026-08-18

Added

Matillion DPC: matillion_dpc.variables config option to supply values for pipeline variables not declared in the YAML.
Matillion DPC: matillion_dpc.environment.default.database config option to resolve the [Environment Default] placeholder.
Matillion DPC: unresolved-variable diagnostics now list the specific missing variable names.

[0.13.1] - 2026-08-12

Fixed

ADF / Synapse connector: fixed source-table resolution for ForEach Copy activities whose source table and SELECT come from @item() config values.

[0.13.0] - 2026-08-11

Changed

Matillion DPC lineage now collapses to two hops (source → output column → target).
Matillion DPC catalog hierarchy changed to Project → Pipeline → Target Table → Column (external ids change; first sync rewrites the repository).

Added

Matillion DPC: Table Update sinks now map target columns; deletes stay table-level.
Matillion DPC: several additional component types now preserve columns (rank, window-calculation, first-last, lead-lag, detect-changes, except, convert-type).
Matillion DPC: unresolved warehouse endpoints are now reported by reason.

[0.12.3] - 2026-07-24

Fixed

Bumped GitPython to >=3.1.52, fixing argument-injection and env-var-exfiltration CVEs (GHSA-2f96-g7mh-g2hx, GHSA-956x-8gvw-wg5v, GHSA-v396-v7q4-x2qj, GHSA-rwj8-pgh3-r573).

[0.12.2] - 2026-07-21

Fixed

Custom-repository connectors now suffix the repository name with the workspace name when targeting multiple workspaces, avoiding name collisions.
dbt connector no longer fails on unknown dbt Cloud enum values (e.g. new remote_backend types).

[0.12.1] - 2026-07-20

Changed

AtScale connector migrated from the deprecated LineageConnector to CustomRepositoryConnector.

[0.12.0] - 2026-07-16

Added

Added the Matillion Data Productivity Cloud (DPC) connector: parses DPL pipeline YAML into Project → Pipeline → Component → Column lineage.

[0.11.20] - 2026-07-15

Fixed

ADF / Synapse connector: fixed cross-linked lineage for config-table ForEach activities with fully parameterized inner Copy steps.

[0.11.19] - 2026-07-07

Added

ADF / Synapse connector: config-table-driven stored procedures now emit one source→target lineage flow per resolved row.
Qlik Replicate connector: endpoints now resolve to Bigeye warehouses via a tiered binder, avoiding ambiguous matches.

[0.11.18] - 2026-07-02

Fixed

ADF / Synapse connector: SQL Server source refs now resolve by schema/table containment instead of requiring a database-name match.
Qlik Replicate connector: same schema/table containment fix applied to SQL Server source endpoints.

[0.11.17] - 2026-06-30

Fixed

ADF / Synapse connector: transient temp tables now collapse correctly across differing dev/prod pool labels.
Qlik Replicate connector: SQL Server source tables now resolve whether the schema is catalogued database-qualified or bare.

[0.11.16] - 2026-06-30

Fixed

Rebuilt on the latest RHEL el10_2 errata, fixing CVE-2026-5450 (glibc) and CVE-2025-5278 (coreutils).

[0.11.15] - 2026-06-25

Changed

Moved the base image from UBI9 (python3.11) to UBI10 (python3.12).

Fixed

Bumped cryptography to >=48.0.1, fixing GHSA-537c-gmf6-5ccf; rebuilt on the latest RHEL errata (openssl-fips-provider).

[0.11.14] - 2026-06-12

Fixed

ADF / Synapse connector: fixed several lineage-graph issues — dangling edges no longer reject the whole sync, cross-activity column edges materialize correctly, database-name overrides apply consistently, and the SQL parser skips uncatalogued-table statements instead of retrying.

[0.11.13] - 2026-06-11

Fixed

Shared REST client: API calls now use a connect/read timeout so stalled connections no longer hang extraction.

[0.11.12] - 2026-06-10

Fixed

ADF / Synapse connector: fixed end-to-end lineage resolution for pipelines using dataset-level sqlPool references, cross-pool stored-procedure lookups, and parameterized linked-service database expressions; added logging for stored-procedure lookups.

[0.11.11] - 2026-06-10

Fixed

ADF / Synapse connector: fixed stored-procedure lineage resolution — case-sensitive name matching, dynamic procedure names, and per-activity build isolation so one failure no longer aborts the whole pipeline.

[0.11.10] - 2026-06-02

Added

ADF / Synapse connector: stored-procedure activities now produce column-level lineage via MetaCenter, and transient temp tables are collapsed to connect real source/target lineage.

[0.11.9] - 2026-05-13

Fixed

Bumped Java 11 to Java 21 to resolve CVEs.

[0.11.7] - 2026-05-13

Added

ADF / Synapse connector: dynamic Tabular Translator expressions are now resolved into concrete column mappings.

Fixed

ADF / Synapse connector: long-running pipeline-history fetches now survive Azure AD bearer-token expiry via automatic re-authentication and retry.
ADF / Synapse connector: nextLink pagination follow-ups now inherit the same retry/timeout/error handling as other API calls.

[0.11.6] - 2026-05-11

Fixed

Qlik Replicate connector: QEM client now retries transient 5xx/connection errors with exponential backoff instead of failing the run.

[0.11.5] - 2026-05-11

Fixed

ADF / Synapse connector: fixed activity-run pagination for Synapse's continuationToken contract (previously only ADF's nextLink was handled).

[0.11.4] - 2026-05-06

Changed

CustomRepositoryConnector.sync() default timeout raised from 2 hours to 8 hours, matching the server-side Temporal activity timeout.

[0.11.3] - 2026-05-05

Fixed

dbt connector: BEGIN; ...; COMMIT; transaction wrappers are now stripped before SQL is sent to the lineage parser.

[0.11.2] - 2026-05-04

Added

Qlik Replicate connector: QEM client now logs proxy resolution details to help diagnose corporate-proxy connectivity issues.

Fixed

ADF / Synapse connector: all HTTP calls now use a connect/read timeout with one retry on read timeout, instead of blocking indefinitely.
ADF / Synapse connector: fixed the Synapse activity-runs API URI (previously used the ADF Management-plane URI for both platforms, causing 404s on Synapse).
ADF / Synapse connector: per-pipeline run-history fetches are now wrapped in a 5-minute watchdog so one stuck pipeline can't block the whole run.
ADF / Synapse connector: API errors now raise a descriptive ConnectorApiError instead of an unhelpful stringified exception.

[0.11.1] - 2026-05-01

Changed

Qlik Replicate connector: Target Column enumeration now starts from the target warehouse's column inventory instead of the source's.
Qlik Replicate connector: column-to-column lineage edges now require a cached column reference instead of falling back to a table-level edge.

Fixed

Qlik Replicate connector: fixed several SAP HANA lineage-resolution issues — backend database resolution, default schema fallback, source-column enumeration retry, and schema-name normalization across warehouses.

[0.11.0] - 2026-04-29

Added

New Qlik Replicate / Qlik Enterprise Manager (QEM) connector.
BIGEYE_TABLE_CACHE_SCHEMA_BATCH_SIZE and BIGEYE_TABLE_CACHE_SCHEMA_BATCH_CONCURRENCY env vars to chunk and parallelize table-cache warm-up requests.

Changed

TableLookupCache.warm_cache now applies schema-batch results all-or-nothing instead of partially loading the cache on a batch failure.

Fixed

Qlik Replicate connector: target-side column lookup no longer falls back to the source schema, avoiding a self-referencing lineage cycle.

[0.10.4] - 2026-04-29

Added

BIGEYE_TABLE_CACHE_PAGE_SIZE env var to control table-cache fetch page size for proxies with shorter request timeouts.
TableLookupCache.warm_cache now retries transport-level failures with backoff.

Fixed

TableLookupCache.warm_cache no longer silently swallows fetch exceptions — failures now abort the run instead of proceeding with an empty cache (which previously caused destructive lineage cleanup).

[0.10.3] - 2026-04-27

Added

dbt: Support for dbt 1.5 manifests, which omit metadata.project_name. New resolve_project_name(manifest) helper in dbt/base/utils.py returns metadata.project_name when set, otherwise infers it from the first model/seed/snapshot node's package_name. Both DBTRepositoryConfigBuilder.build() and DBTConnector.create_monitors() apply the resolution during their parse loops, write the inferred value back onto manifest.metadata.project_name, and log a warning naming the file path and dbt version. Manifests where inference also fails (e.g., no qualifying nodes) are skipped with a warning instead of crashing.

Fixed

dbt: Failed to sync for workspace ...: sequence item 1: expected str instance, NoneType found crash when a manifest's metadata.project_name was null/missing. The previous ternary manifest.metadata.project_name if manifest.metadata else "dbt" only handled a missing metadata object; a present-but-null project_name (declared Optional[str] on DbtMetadata) flowed as None into _build_external_id's "__".join(...). Resolved by the dbt 1.5 inference described above.
dbt: TableLookupCache.warm_cache now dedupes schema_ids before calling get_tables_post. _schema_cache is keyed by lowercased fq_name, so callers passing case-variant fq_names (common across multi-project dbt syncs) previously appended the same schema_id 2–3× in a single request body. Switched the local accumulator from list to Set[int] and sorted before sending for deterministic request bodies.

[0.10.2] - 2026-04-23

Added

dbt Cloud: extract caching. When the same dbt Cloud run_id is still the latest run for a job, the connector now reuses the previously downloaded manifest.json, catalog.json, run_results.json, and run/*.sql artifacts instead of re-downloading them. **Cache is enabled by default** (dbt.cache.enabled=true);
dbt Cloud: parallel project extraction within an account. New dbt.parallelism property (default 4) bounds the thread pool used to process projects concurrently.

Changed

dbt Cloud: Failed downloads of catalog.json now log at WARN instead of ERROR.
Framework: BaseIntegrationClient now retries automatically on HTTP 429 (rate limited). Retries honor the Retry-After header when present (capped at 60s), otherwise use exponential backoff with ±20% jitter (1s → 2s → 4s → 8s → 16s, capped at 30s) up to 5 retries.
Framework: Connector.sync() default timeout raised from 1200s (20 min) to 7200s (2 hours).

Fixed

dbt Cloud: cross-project cache pruning. When multiple projects shared the same dbt.local.manifests.location, each project's prune step would delete the other projects' current run_id directories. Pruning is now performed once at the end of extraction against the union of latest run IDs across all accounts and projects, so shared-cache setups behave correctly.

[0.10.1] - 2026-04-23

Added

BIGEYE_FILE_OUTPUT_ROOT environment variable (also settable as file.output.root in settings.properties) that overrides the base directory for connector-generated files. Defaults to the current working directory; the directory is created if missing. Affects: extracts/<connector>/<env> directories (dbt, Looker, AtScale, ADF, S3, ADLS, SharePoint), debug_configs/ written under --save-config, the default run_summary.json location for both run and monitor commands, and the lineage_template.yaml paths under S3/ADLS/SharePoint. An explicit --summary-output path still wins over the env-var-derived default.

Fixed

dbt Cloud: GET /api/v3/accounts/{id}/projects/ no longer fails with pydantic validation errors when the API returns values outside our internal enums. DBTJobSchedule.date/time, DBTJob.job_type, DBTJob.state, DBTRepository.state, DBTJobTriggers.custom_branch_only, and JobSchedule.date now accept any string/int/None value returned by the API. Affected DBTListProjectsResponse parsing for projects with jobs whose job_type was scheduled/other, schedule discriminators like days_of_week/at_exact_hours, or null custom_branch_only — previously these caused the entire account's extraction to fail.

Security

Bump cryptography to >=46.0.7,<47.0.0 to resolve [CVE-2026-39892](https://nvd.nist.gov/vuln/detail/CVE-2026-39892) (CVSS 9.8 — buffer overflow when a non-contiguous Python buffer is passed to cryptography APIs such as Hash.update()). Affected the bigeyedata/source-connector image via the previous 46.0.6 lock.

[0.10.0] - 2026-04-06

Added

ADLS Gen2 support

[0.9.4] - 2026-04-03

Added

SQL parser: Exponential retry policy on /api/v1/sql-parser requests. On 500 errors, retries up to 4 times with exponential backoff (1s, 2s, 4s, 8s). Non-500 errors are not retried. If all retries fail, logs a warning and continues processing with empty lineage for that query.
dbt: Connector now loads compiled SQL from run/<package>/<path>/*.sql files, falls back to run_results.json compiled_code, then manifest.json compiled_code.
dbt Cloud: dbt.runs.last.x.days config option to control how far back to query dbt Cloud runs (default: 7). The connector still selects only the latest successful run per job, but this allows finding runs older than 7 days when needed.

Fixed

SQL parser: Fixed case-sensitive table name matching that caused output tables with mixed-case names (e.g., from dbt manifests) to not be found in the Bigeye catalog.
SQL parser: Fixed column-level lineage for Snowflake variant field extraction (SOURCE_DATA:FIELD::string as FIELD). Alias nodes from the SQL parser graph are now matched to output table columns.
dbt: Removed passthrough fallback that created incorrect column-level lineage edges and phantom columns (e.g., SOURCE_DATA) on dbt models. Models without deterministic column-level lineage now produce no lineage edges instead of incorrect ones.

[0.9.3] - 2026-04-02

Added

ADF connector: Azure Synapse Analytics pipeline support.

[0.9.2] - 2026-04-01

Fixed

dbt Cloud: Trigger cause filter now works correctly. Added include_related parameter to the List Runs API call so that trigger fields are populated in responses. Previously, the dbt.trigger.cause filter silently matched zero runs.

Added

dbt Cloud: dbt.environments.type config option to filter runs by dbt environment type (deployment, development). Use this to include only deployment environments and exclude development environments.
dbt Cloud: dbt.environments.deployment.type config option to filter runs by dbt deployment type (production, staging). Use this to target production or staging deployment environments specifically.
dbt Cloud: dbt.environments.include and dbt.environments.exclude config options to filter runs by environment name. Allows separating production runs from QA/staging runs within the same project. Uses substring matching, consistent with existing account/project/job filters.
Run summary: Brief human-readable summary printed to stdout after each run/monitor command completes, showing status, duration, node/edge counts, and any errors or warnings per environment and workspace.
Properties: Global base properties support. Properties defined without a .N numeric suffix are now treated as base/global defaults applied to all indexed connections. Indexed properties (.1, .2, etc.) override base values when present.
Integration tests: End-to-end dbt connector integration tests that trigger a real dbt Cloud job, extract artifacts, sync to Bigeye staging, and validate catalog nodes, column structure, and column-level lineage paths.

[0.9.1] - 2026-03-30

Fixed

PowerBI supplemental connector: Fix missing lineage from additional sources to reports caused by passing data_node.id
(lineage ID) instead of node_entity_id (entity ID) to the catalog entity children API in the report resolver

[0.9.0] - 2026-03-30

Added

ADF: SQL parser now discovers source table schemas from FROM/JOIN clauses in SQL queries, enabling upstream column-level lineage for Script activities with 2-part table names (e.g., replica_crossreference.DvmValue)
ADF: SQL parser treats SQL Server and Synapse warehouse types as compatible, so Synapse sources are found when pipelines reference SQL Server linked services
ADF: MERGE INTO statement support for extract_insert_into_target and extract_insert_select_column_mappings, enabling lineage for pipelines that use MERGE-based data loads
ADF: CASE WHEN source column extraction — _parse_select_columns extracts the THEN column from MAX(CASE WHEN ... THEN v.Value END) AS Alias patterns, mapping all output columns to their actual source column
ADF: ForEach iteration resolution Strategy 3 — when Lookup output and ForEach input are empty in run history, resolves iteration items from inner Script activity run inputs
ADF: Table cache fallback for upstream lineage — when the backend SQL parser fails (400/500), source columns are resolved directly from FROM/JOIN tables via the Bigeye warehouse table cache
ADF: stored_procedure_name field on ADFActivity now accepts ADF expression dicts (e.g., {'value': '@item().StoredProcedure', 'type': 'Expression'})
ADF: SQL Server bracket-quoted identifier support in SQL parsing ([Location] aliases, [transaction] table names)
ADF: resolve_database_for_schema method on table cache for suffix-matching 2-part schema names to their full database.schema form
ADF: Data flow column node deduplication — prevents duplicate Column nodes when multiple sinks share column names

Fixed

ADF: SELECT DISTINCT and SELECT ALL queries now parse correctly for column mapping extraction
ADF: Parenthesis-aware column splitting prevents incorrect splits on commas inside function calls (e.g., CONVERT(VARCHAR(32), Hashbytes(...)))
ADF: Script activity downstream edges (ADF column -> target warehouse column) are now created before the SQL parser call, so they survive backend parser failures
ADF: ForEach iteration naming uses placeholder default_database for 2-part INSERT INTO targets, producing descriptive names like Sql Query [CR_EIACOMGRP] instead of numeric indices
ADF: Database name overrides are now applied to the output table before passing to the SQL parser, ensuring schema lookups use the correct environment-specific database name
ADF: ADFPipelineRun.message field handles explicit null values from the API without validation errors

[0.8.0] - 2026-03-30

Added

Support for Microsoft SharePoint

[0.7.1] - 2026-03-23

Added

Proxy authentication support for connector HTTP clients via settings.properties (proxy.auth.user, proxy.auth.password, auth.header.key) or environment variables (BIGEYE_PROXY_AUTH_USER, BIGEYE_PROXY_AUTH_PASSWORD, BIGEYE_AUTH_HEADER_KEY)
Configurable logging via settings.properties (log.level, log.file) or environment variables (BIGEYE_LOG_LEVEL, BIGEYE_LOG_FILE)
--settings CLI flag to specify a settings.properties file for cross-cutting configuration
dbt Cloud: dbt.trigger.cause config option to filter job runs by trigger cause (e.g. Schedule, GitHub). When set, only the latest run whose trigger cause contains the specified value (case-insensitive) is selected per job. Defaults to latest run regardless of trigger cause when not set.

Fixed

dbt Cloud: Use v2 API for runs and jobs endpoints (v3 does not support these resources)
dbt Cloud: Use Accept: */* header for artifact downloads to fix 406 errors on cell-based hosts (e.g. *.us1.dbt.com)
dbt Cloud: Use custom auth header for Bearer token in proxy environments instead of hardcoded Authorization

Updated

dbt Cloud client pagination support for all list endpoints
dbt Cloud client uses per-instance headers instead of shared module-level headers

[0.7.0] - 2026-03-18

Added

PowerBI Supplemental connector for additional PowerBI sources not supported by Lineage Plus

[0.6.0] - 2026-03-16

Added

Initial support for Azure Data Factory

Updated

Resolve environment variable references in property files (e.g. ${BIGEYE_API_KEY})

[0.5.1] - 2026-03-10

Updated

Configurable lineage granularity for S3 connector (s3.lineage.granularity=folder|file)
Auto-deploy support for S3 lineage templates when all edges are resolved
Migrated S3 connector to CustomRepositoryConnector
Folder mode derives folder names from lineage mapping prefixes

Fixed

S3 s3_uri no longer doubles the bucket name in paths
table_fqn resolution for lineage edge sync

[0.5.0] - 2026-03-04

Added

New CustomRepositoryConnector base class that consolidates catalog creation and lineage sync into a single declarative configuration, replacing the legacy multi-step LineageConnector workflow

Updated

Migrated DBTConnector from LineageConnector to CustomRepositoryConnector

Deprecated

LineageConnector base class in favor of CustomRepositoryConnector
SimpleSqlParserResponse.to_lineage_configurations() in favor of to_repository_sync_edges()

Removed

DBTLineageController (replaced by DBTRepositoryConfigBuilder)

[0.4.4] - 2026-02-13

Updated

SQLParser no longer requires the use of CTEs or temp tables in order to retrieve lineage for dbt

[0.4.3] - 2026-01-30

Added

Support for custom webhook URL and webhook header notifications in the DBTConnector for external monitoring

[0.4.2] - 2026-01-29

Fixed

Fixed path duplication for catalog.json files in the DBTConnector

[0.4.1] - 2026-01-21

Updated

DBTConnector supports execution level statuses in addition to test level statuses

[0.4.0] - 2025-12-04

Added

Support for external monitoring of dbt connector

[0.3.0] - 2025-11-26

Added

Initial implementation for the Looker Connector

[0.2.4] - 2025-11-18

Fixed

AtScale connector properly handles select star queries
Various improvements and bugfixes to the AtScale connector

[0.2.3] - 2025-11-14

Fixed

DBT Connector now correctly pulls the last successful run for each job, not just the last successful run for all jobs

[0.2.2] - 2025-11-12

Fixed

Fixed docker image dependencies not getting installed

[0.2.1] - 2025-11-12

Fixed

Fixed docker image dependencies not getting installed

[0.2.0] - 2025-11-07

Added

Support for AtScale catalog generation and lineage

[0.1.0] - 2025-11-07

Initial release