Lineage Plus Agent

The Lineage Plus agent collects lineage metadata from your databases, BI tools, and ETL platforms. Unlike the Data Source agent, Lineage Plus runs on-demand or on a schedule rather than as a continuous service.

Pre-Install Checklist

1. VM / Host — Ubuntu 20.04+ or RHEL 8+ preferred

   • 25 GB RAM
   • 4 CPU
   • 55 GB disk space

2. Networking

   • Firewall access to app-metacenter-portal.bigeye.com and app-metacenter-solr.bigeye.com
   • Firewall rules must not strip Authorization headers
   • Egress to your data sources and Bigeye SaaS environment
   • Egress to the licensing endpoint: https://caecsxb9bk.execute-api.us-west-2.amazonaws.com
   • Access to pull images from docker.io

3. Credentials from Bigeye

   • Company name associated with your agent installs
   • Lineage Plus license key
   • Docker PAT

Installation

# Run the installer and select LINEAGE_PLUS
./bigeye-agent install

# Add connectors for lineage collection
./bigeye-agent add-connector

Commands

Run Lineage Collection

# Run all configured connectors
./bigeye-agent lineage run

# Run a specific connector type
./bigeye-agent lineage run -c snowflake

# Run multiple connector types
./bigeye-agent lineage run -c snowflake -c tableau

# Run a specific named connection
./bigeye-agent lineage run -c snowflake -n my-snowflake-prod

# Skip re-syncing configuration before running
./bigeye-agent lineage run --no-sync

# Skip email notifications for this run
./bigeye-agent lineage run --skip-notifications

If the connector runs successfully, you'll see a success message. Allow several hours for lineage data to sync into Bigeye.

Update Configuration

# Re-enter Lineage Plus configuration
./bigeye-agent lineage configure

# Overwrite existing configuration
./bigeye-agent lineage configure --overwrite

Configure Notifications

./bigeye-agent lineage configure notifications

Configures email notifications sent after lineage runs complete. You'll be prompted for:

1. Enable/disable notifications — Master toggle for email notifications.
2. Email recipients — Comma-separated list of email addresses.
3. Notify on success — Whether to include successful runs (default: Yes).
4. Notify on failure — Whether to include failed runs (default: Yes).

YAML Configuration

agents:
  - type: LINEAGE_PLUS
    properties:
      notifications:
        enabled: true
        recipients:
          - [email protected]
          - [email protected]
        notify_on_success: true
        notify_on_failure: true

Notification emails include a subject line ([Lineage Plus] N connections - X Success, Y Failure), a summary, a results table for each connection, and error details for any failures.

Configure Log Retention

./bigeye-agent lineage configure log-retention

Logs are stored in ./logs/<connector_type>/ in timestamped directories. By default, the 3 most recent log directories are retained per connector type.

Enter retention settings as connector:count pairs:

sqlserver:10, powerbi:5, tableau:7

YAML Configuration

agents:
  - type: LINEAGE_PLUS
    properties:
      log_retention:
        sqlserver: 10
        powerbi: 5
        tableau: 7

Connectors not listed use the default retention of 3 directories. After each run, old log directories beyond the retention count are automatically removed.

Update Connections

# Preview connection mapping changes
./bigeye-agent lineage update-connections

# Apply the updated mappings
./bigeye-agent lineage update-connections --apply

# Process a specific connection only
./bigeye-agent lineage update-connections -cn my-snowflake

# Skip config validation
./bigeye-agent lineage update-connections --skip-config-validation

Upgrade

./bigeye-agent lineage upgrade

Updates the Lineage Plus agent to the latest Docker image version.

View Logs

./bigeye-agent lineage logs -c snowflake

Share Diagnostics

./bigeye-agent lineage share-diagnostics -c snowflake
./bigeye-agent lineage share-diagnostics -c snowflake -m "Failing on large tables"