Agent Connection
Bigeye Agent Installation Guide
Bigeye provides a Docker image for hosting the Bigeye Data Source Agent on your infrastructure. The agent may be deployed as a standard Docker container instance using tools such as docker run
or docker-compose
. You may also use container orchestration systems such as Kubernetes or Nomad.
Requirements
Pre-Install Checklist
You will need the following information to install the Bigeye agent.
- Bigeye credentials
- You will need a Bigeye account.
- The user must be an Admin or an Owner role in Bigeye.
- Company prefix
- This is the first part of your Bigeye URL (e.g. “app” for app.bigeye.com)
- Company UUID
- Bigeye will provide this.
- docker access token
- Bigeye will provide this
- Set up a VM / host (infrastructure requirements below)
Infrastructure Requirements
- Host with docker installed and running
- Other Docker platforms such as AWS ECS, or orchestration systems such as Kubernetes will work, but their installation is not directly supported by Bigeye.
- Install docker if it is not already intalled [CentOS][RHEL][Debian][Ubuntu]
- NOTE for RHEL, the official docker instructions don't quite work. Using docker's CENTOS yum repository instead is a good workaround.
sudo yum install -y yum-utils sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo sudo yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin sudo systemctl start docker # test it sudo docker run hello-world
- You can test if docker is running by using
docker info
and getting a non-error response
docker compose
should already work if you've installed docker with the above instructions. If not, Install docker compose (Instructions- Run
docker compose version
to test if it is installed correctly. The version should be 2.0 or higher.
- Run
- Minimum HW size: 4 CPU, 8GB mem (ie AWS t3.xlarge, GCP e2-standard-4, Azure B4/D4, etc)/
- 25GB disk space on /var is recommended
- Network access for agent subnets
- Access to the data sources you wish to add to Bigeye for monitoring
- Access to the Bigeye Saas environment
- Access to pull the agent image from docker.io
Install
Docker compose is a docker native tool that is used to define docker containers as yaml configuration instead of having to manage them on the command-line with flags etc. It will be used heavily for installing and running the agent.
Create docker-compose.yaml
Download bigeye-agent-docker-compose.yaml as docker-compose.yaml and pull the latest agent image.
docker logout
docker login --username bigeyedata --password <docker access token>
wget https://bigeye-public-web.s3.amazonaws.com/bigeye-agent-docker-compose.yaml --output-document=docker-compose.yaml
docker compose pull
Anytime you wish to install the latest version of the agent, docker compose pull
can be run again to pull the latest image.
If docker compose is not able to be installed, the native docker commands are in the Appendix and can be used instead.
If docker pull is not possible on the system where you will be running the agent, there are instructions on how to import/export the container from a test system instead in the Appendix.
Configuration
1. Generate agent encryption keys and configuration
The agent requires a configuration file and two X509 certificate pairs to start. We’ll present those to the agent container by means of a Docker bind volume attached to /app/config
on the container. Here we will run the setup script to create the configuration file and certificates for the agent to use.
Note: Fill in your company UUID in the below docker command
mkdir agent_config
# Run setup script
# <company_uuid> - Bigeye will provide this
# -p app - single-tenant saas customers will use something different (ask Bigeye)
docker run --rm -it -v "${PWD}/agent_config:/app/config" --entrypoint setup_agent.sh docker.io/bigeyedata/agent:latest -p app -e -c <company UUID>
2. Install Bigeye mtls keys
You will also need to add mTLS certs which will be provided by Bigeye. Copy the certs provided into the agent_config
directory, be sure to keep the name of the files the same.
3. Add data sources to the agent
Edit agent_config/agent.yaml
and add each source. For each source you will need:
- Warehouse type (MySQL, Oracle, Redshift, etc)
- Hostname or IP address
- Port number
- Username
- Password (double quotes in the password should be escaped with a , ie
"my_pass\"_with_a_double_quote"
) - Database name
Running the Agent
Note: All configuration steps above must be complete before deploying the agent.
# The -d flag (detatch) runs the container in the background
docker compose up -d
# Check that the container is running
docker ps -f name=bigeye-agent
# View container logs (-f can be used to “tail” the logs)
docker compose logs
# Stopping the agent if desired
docker compose down
To confirm that the container started successfully, look for a line in the Docker log similar to the following:
INFO Agent started successfully in 4076ms. Polling for activities...
Note that you can launch more instances of the Agent on the same machine by copying the bigeye-agent block in docker-compose.yaml
(be sure to name it something else ie bigeye-agent-xxx).
Add data sources in Bigeye UI
Log into the Bigeye UI ie https://app.bigeye.com for multi-tenant saas or https://.bigeye.com for single-tenant saas.
- Go to Settings -> Advanced Settings -> Sources tab -> Add Source button
- Select data source type (ie Snowflake)
- Choose “Connect with in-VPC agent
- The Name field in the next screen much match exactly the Identifier field that was used in agent.yaml
- The rest of the steps beyond this will be the same as a directly connected data source (non-agent). https://docs.bigeye.com/docs/getting-started#1-connect-your-sources
Troubleshooting
Common agent errors in agent log
javax.net.ssl.SSLException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
- Description: Certificate given to Bigeye in “Step 2. Upload agent authentication key” does not match.
- Fix: Work with Bigeye support to resolve this (
config/mtls_ca.pem
md5 should be checked and in Bigeye’s workflow service)
Deadline exceeded (DEADLINE_EXCEEDED)
- Description: Network connectivity between the Bigeye agent and the Bigeye workflow service does not work
- Fix:
- Check that workflowUrl is set correctly in
./config/agent.yaml
- Check that the port is open, ie nmap -Pn -p 443
- Nmap should return “open” as the state of the port and not “filtered”
PORT STATE SERVICE 443/tcp open https
- Find out why network connectivity is not working and resolve
- Check that workflowUrl is set correctly in
Exception in thread "main" io.grpc.StatusRuntimeException: CANCELLED: RST_STREAM closed stream. HTTP/2 error code: CANCEL
- Description: Something in the networking infrastructure is closing the connection between the agent and Bigeye SaaS.
- Fix: Work with your security / networking team to find where this is happening. Something that helps narrow things down greatly is to prove the agent and configs work from outside of your production network. IE start the agent with the same configuration files on a laptop at home/office (somewhere not in the production network).
jdbc timeout / error reaching host:port
- Description: Networking connectivity between the agent and the data source is not set up.
- Fix: Likely culprits are firewalls or subnets that do not have a route to reach the subnet that the data source is running on
Appendix
Import Image Archive
If docker pull can not be used on the production system where the agent will be run, use the following instructions to save and transfer the docker image from a test system / local workstation.
save the agent image as a file
Follow the docker compose pull instructions on a test system to pull the agent image. Then export the image to a file using docker save.
# Export image to file
docker save docker.io/bigeyedata/agent:latest > bigeye_agent_$(date '+%Y-%m-%d').tar
import the agent image on the production system
Copy the image archive file to the system where the agent will be run. Import it onto the host that will be running the agent using the commands below.
# Import image
docker load -i < Path to image archive here >
# Verify the image was loaded successfully
docker image ls bigeyedata/agent:latest
Running without docker compose
It's possible to use docker run directly if a newer version of docker that has docker compose is not available.
# “docker logout” is required first if you are already logged in as a different account
docker login --username bigeyedata --password <docker access token>
docker pull docker.io/bigeyedata/agent:latest
docker run -d --name bigeye-agent -v ${PWD}/agent_config:/app/config/ --restart always docker.io/bigeyedata/agent:latest
docker logs bigeye-agent
AWS Secrets manager
The agent can read your data source password from AWS Secrets Manager instead of having the password in plain text in agent.yaml
. To use it, use the following example snip below in your agent.yaml. The aws credentials are used by the agent to read the secret from AWS Secrets Manager.
# The AWS access key and secret key are used by the agent to read from AWS secrets manager
awsAccessKey: AKIAIOSFODNN7EXAMPLE (ie what normally goes in AWS_ACCESS_KEY_ID)
awsSecretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY (ie what normally goes in AWS_SECRET_ACCESS_KEY)
region: us-east-1 (AWS region where AWS secrets manager secret resides)
sources:
- identifier: some snowflake db1 using aws secrets manager for password
connectionFactory:
type: snowflake
host: mycompany.snowflakecomputing.com
port:
user: user
# AWS Secrets Manager secret name
asmSecretNameForPassword: my/aws/secret_manager/secret_name
databaseName: somedb1
Environment variable for password
The agent.yaml can accept environment variables in place of the plain text. This is useful for things like data source credentials. Update agent.yaml
to use an environment variable for the user and password like shown below. The syntax is specific, do not wrap the variable in quotes: ${good} "${bad}"
# Example data source using environment vars for credentials
- identifier: db with env vars for creds
connectionFactory:
type: snowflake
host: mycompany.snowflakecomputing.com
port: 443
user: ${SECRET_USERNAME}
password: ${SECRET_PASSWORD}
databaseName: somedb
Then update docker-compose.yaml
to pass through the environment variables (last 3 lines):
version: "3.9"
services:
bigeye-agent:
image: "docker.io/bigeyedata/agent:latest"
restart: always
volumes:
- "${PWD}/agent_config:/app/config"
environment:
- SECRET_USERNAME
- SECRET_PASSWORD
Write a script that can retrieve the credentials
Finally write a script that can retrieve secrets from your password keeper (HashiCorp Vault, Azure Key Vault, etc) and use that to populate the environment variables
export SECRET_USERNAME=$(username_script.sh)
export SECRET_PASSWORD=$(password_script.sh)
docker compose up
(Optional) Automate
There are many ways of automating this. A low effort way would be to put the variable exports into your .bashrc or .bash_profile so the environment variable is populated on login, then you don't have to remember to set it before running docker compose up
.
Updated 2 days ago