Agent CLI

The Bigeye Agent Installer is a CLI tool to help Bigeye customers install our Data source, Lineage Plus, and Datahealth agents. The Bigeye Agent Installer will walk customers through an end-to-end installation process of our agents and validate information is entered correctly. The purpose of this CLI tool is to help expedite the process of installing Bigeye agents.

Installing Agents

Agent Infrastructure Requirements

‭Our agents can be run on a single virtual machine or can be managed separated on their own. Having both agents together‬ can make them easier to find but can also make it harder to diagnose issues related to each‬ agent.‬

Single Virtual Machine or Server Configuration‬

If you would like to run both the Data Source and Lineage Plus agents on a single machine, here are the recommended specifications.

Ubuntu (20.04+) or Redhat Linux (RHEL 8+) preferred‬‭
32 GB memory‬
4 cores‬
80 GB disk space

‭Two Virtual Machine or Server Configuration‬

If you would like to run the Lineage Plus and Data Source agent on separate machines, then here is the recommended sizing for each.

Lineage Plus agent:‬

Ubuntu (20.04+) or Redhat Linux (RHEL 8+) preferred‬‭
25GB of RAM‬
‭4 CPU‬
‭55 GB disk space‬

Data Source agent:‬

‭8GB RAM (ie AWS t3.xlarge, GCP e2-standard-4, Azure B4/D4, etc)‬
4 CPU‬
25GB disk space on /var is recommended‬

Networking Requirements

Firewall access to the hostname and URL paths provided below:
- app-metacenter-portal.bigeye.com‬
- app-metacenter-solr.bigeye.com‬
The firewall rules should NOT strip any Authorization headers for the below mentioned‬ host/domain names.
‭Port 443/80 open for the host name app-metacenter-rmi.bigeye.com
Egress (outbound) Access to the data sources you wish to add to track Lineage in Bigeye
Egress (outbound) Access to the Bigeye SaaS environment
- app.bigeye.com
Ingress (inbound) Access to retrieve Licenses from Bigeye
- https://caecsxb9bk.execute-api.us-west-2.amazonaws.com
Access to pull images from docker.io

Setting up the Agent using the installer

The Bigeye Agent Installer is hosted in a public s3 bucket in AWS, it can be downloaded using:

# Ubuntu
curl -O https://bigeye-public-web.s3.us-west-2.amazonaws.com/bigeye-agent-installer/latest/linux/bigeye-agent

# RHEL
curl -O https://bigeye-public-web.s3.us-west-2.amazonaws.com/bigeye-agent-installer/latest/rhel/bigeye-agent

# Windows
curl -O https://bigeye-public-web.s3.us-west-2.amazonaws.com/bigeye-agent-installer/latest/windows/bigeye-agent.exe

# Mac
curl -O https://bigeye-public-web.s3.us-west-2.amazonaws.com/bigeye-agent-installer/latest/mac/bigeye-agent

Once the Bigeye Agent Installer has been downloaded, you have to unzip and grant permissions to the executable.

# Extract the archive
tar -xf bigeye-agent

# Update permissions
chmod +x bigeye-agent 

# Windows - no step needed

Now you can access the CLI commands. There are a variety of helper commands that can be run.

# View available commands
./bigeye-agent --help 

# Main installation command that will walk through all agent installations
./bigeye-agent install

The Bigeye Agent Installer CLI tool will require some information about your Bigeye instance Please ensure you have the following references available. If you do not have these, then contact your Bigeye Customer Success representative.

Name	Example	Description
Bigeye base url	https://app.bigeye.com	The url of the Bigeye service, defaults to https://app.bigeye.com
Personal API Key	bigeye_pak_123fakepassword	The personal API key for an Admin level user in Bigeye. Can be generated in Bigeye (see docs here https://docs.bigeye.com/docs/using-api-keys)
Agent API Key	bigeye_agent_123fakepassword	The agent API key, these are API keys associated to the company not an individual user. Can be generated in Bigeye (seee docs here https://docs.bigeye.com/docs/using-api-keys)
Docker PAT	dckr_pat_abc123	The docker personal access token provided by Bigeye used to pull bigeye images from Docker Hub.
Company Name	Demo Install	The company name associated with your agent installs. This will be provided by Bigeye.
License Key	abc123examplekey	The password to authenticate to your tenant and get your associated Lineage Plus License. This will be provided by Bigeye.

Adding connections

Once the installation process has completed, you can move onto the steps to add and run your connectors and begin collecting lineage for Bigeye.

# Add connectors
./bigeye-agent add-connector

# Add a specific type of connector (optional)
./bigeye-agent add-connector -c snowflake
./bigeye-agent add-connector -c tableau
./bigeye-agent add-connector -c databricks
./bigeye-agent add-connector -c powerbi
./bigeye-agent add-connector -c databricks

If you need to edit your existing connections, the easiest thing to do is to modify the bigeye_agent.yml file directly. This file contains all the connection information and configuration items for your agents. After editing this file, you will need to re-generate the config files for each of your configured agents. To do that you can run the ./bigeye-agent sync command to re-generate the necessary config prior to running any connectors.

Once connection information has been added and validated by the installer the next step is to run the various agents.

Running the Lineage Plus Agent

Once connectors have been successfully configured in your Lineage Plus agent, you can leverage the Bigeye Agent CLI tool to run your connectors and begin collecting lineage. The run command can be used to run the connectors and will output the logs to your console.

# Run a Snowflake connector
./bigeye-agent lineage run -c snowflake

# Run a SQL Server connector
./bigeye-agent lineage run -c sqlserver

# Run a Tableau connector
./bigeye-agent lineage run -c tableau

# Run Power BI connector
./bigeye-agent lineage run -c powerbi

If the connector ran successfully you will see a success message in the console indicating that the run has completed. After connector has been run, please allow several hours before this can be synced into Bigeye. If the connector failed to run, then you will be given a failed message a long with the list of errors from the run. Please try to correct those errors and re-run.

Running the Data Source Agent

The Data Source agent is a always running service. Once you have defined the connections you can start the Data Source agent using the following command.

# Start the data source agent
./bigeye-agent source start

# Stop the data source agent
./bigeye-agent source stop

# Review the logs of the data source agent
./bigeye-agent source logs

# Upgrade to the latest image for the Data Source agent
./bigeye-agent source upgrade

Troubleshooting

[PYI-13495:ERROR] Failed to load Python shared library '/tmp/_MEID1H6U6/libpython3.10.so.1.0': dlopen: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /tmp/_MEID1H6U6/libpython3.10.so.1.0)

Description: The binary produced by pyinstaller depends on GLIBC and the version of GLIBC it depends on is based on the OS that the executable gets generated. This error indicates the the version of GLIBC used is not supported by the OS.
Fix: Get a VM with a newer OS or request if Bigeye can build the executable to meet the needs of your exact VM.

jdk.JdkError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)>

Description: The Python environment is unable to verify the SSL certificate of the server you're trying to connect to.
Fix: Python might not automatically use the updated certificates. You can ensure Python uses the system's CA certificates by setting the SSL_CERT_FILE environment variable. This will tell Python to use the system's certificate bundle for SSL verification.

export SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt

(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))