Generating Lineage

Automatic Lineage Generation

Bigeye can automatically detect lineage from Snowflake, Redshift, BigQuery, and Tableau sources. We do this by inspecting the source metadata for these connection types, and building a graph of relationships between tables in that source based off of that data.

📘

Lineage permissions

For lineage to automatically gather, please make sure your Bigeye user has the most up to date permissions listed in our Data Sources connection guides. Lineage does not have specific permission sets, it relies on the same permissions as our popularity scores and table based metrics.

At present, our automatic lineage detection will create table level connections, so you can easily see which of the tables in your source depend on/write into one another. Please note - as of our initial release, views are excluded from automatic detection as they do not have entries in the source metadata. We are working to add support for views in an upcoming release - keep an eye out for updates!

Lineage collection is scheduled to run roughly every 24 hours (similar to our catalog indexing), but you can force an update on any given source by running: /api/v1/lineage/sources/:id/lineage. See our lineage API documentation for more details.

Lineage is also connected to our indexing processes, so you can trigger lineage generation by clicking 'rescan schema' or 'rescan table' on any catalog page.

API-based Lineage Generation

For sources where automatic lineage collection is not currently supported, or to specify any relationships that are not listed in the information schema, you can use our lineage API to define your own table connections. Once generated, all other lineage features will be available for any tables with connections defined.