Snowflake Lineage Connector

The Snowflake Lineage Connector allows Bigeye to track query history and lineage from your Snowflake instance. This connector requires appropriate permissions and configuration to access query history and metadata.

Prerequisites

Before configuring the connector, ensure you have a Snowflake user with the required permissions.

Note: See the Snowflake setup guide for detailed instructions on creating users and granting the necessary permissions.

Connector Parameters

The Snowflake Lineage Connector requires the following parameters in addition to standard database connection parameters:

Parameter	Description
`db.connection.url`	JDBC URL for connecting to your Snowflake instance.
`db.connection.driver_class`	JDBC driver class name for Snowflake.
`db.connection.username`	Snowflake username for the connector.
`db.connection.password`	Password for the Snowflake user.
`db.catalog.name.include`	Optional list of catalogs to include.
`db.catalog.name.exclude`	Optional list of catalogs to exclude.
`db.snowflake.history.schema`	Schema name where query history tables will be stored.
`db.snowflake.history.table`	Table name for storing query history.
`db.snowflake.query.history.incremental.load`	Set to `true` to load query history incrementally.
`db.snowflake.invoke.query.history`	Set to `true` to invoke Snowflake query history API.
`db.snowflake.history.run.period`	Periodicity for populating the history table (e.g., `1d`, `1h`).
`db.snowflake.history.exclude.catalog`	Optional list of catalogs to exclude from history tracking.

Example Configuration

# Snowflake connection
db.connection.url=jdbc:snowflake://<account>.snowflakecomputing.com
db.connection.driver_class=net.snowflake.client.jdbc.SnowflakeDriver
db.connection.username=your_user
db.connection.password=your_password

# Optional catalog filters
db.catalog.name.include=SALES,MARKETING
db.catalog.name.exclude=TEST

# Query history configuration
db.snowflake.history.schema=BIGEYE_HISTORY
db.snowflake.history.table=QUERY_HISTORY
db.snowflake.query.history.incremental.load=true
db.snowflake.invoke.query.history=true
db.snowflake.history.run.period=1d
db.snowflake.history.exclude.catalog=INTERNAL