Guides
Start typing to search…

Docker Deployment

Deploy Bigeye agents using Docker containers

Docker is the default deployment method for all Bigeye agents. The Agent CLI manages Docker containers for each agent type, handling image pulling, container configuration, and lifecycle management.

Prerequisites

  • Docker installed and running
  • Docker PAT provided by Bigeye (for pulling agent images)
  • Sufficient disk space for Docker images and container volumes

How It Works

When you run agent commands (e.g., source start, lineage run), the Agent CLI:

  1. Pulls the latest Docker image using your Docker PAT
  2. Creates a container with the appropriate volume mounts, environment variables, and network configuration
  3. Starts the container (attached or detached depending on the agent type)

Configuration files generated by the CLI are mounted into containers as volumes so agents can read your connector settings.

Agent Deployment Patterns

Long-Running Services

These agents run as persistent Docker containers:

  • Data Source agent — ./bigeye-agent source start / stop
  • Cross-Source agent — ./bigeye-agent crosssource start / stop
  • SDS agent — ./bigeye-agent sds start / stop (runs two containers: Java + Python)

On-Demand Execution

These agents run as short-lived containers that exit after completing:

  • Lineage Plus — ./bigeye-agent lineage run -c <connector>
  • DataHealth — ./bigeye-agent datahealth run --file-name <file>
  • External Monitors — ./bigeye-agent external-monitors run -c <connector>

SDS Agent: Dual Container Architecture

The SDS agent is unique in that it runs two interconnected containers:

  1. Java agent — Connects to data sources and collects data samples
  2. Python service — Performs classification on the collected data

The Python service shares the Java container's network stack and is configured with a 12 GB memory limit. Both containers are started and stopped together using the sds start and sds stop commands.

Proxy Configuration

When proxy settings are configured in bigeye_agent.yml, the CLI automatically:

  • Sets HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables in containers
  • Configures Java proxy options (-Dhttp.proxyHost, -Dhttps.proxyHost, etc.) for Java-based agents
  • Passes proxy credentials if configured

Docker Sudo

If Docker requires sudo on your system, set docker_sudo_required: true in your bigeye_agent.yml configuration. The CLI will prepend sudo to all Docker commands.

Upgrading Agent Images

Each agent type has an upgrade command that pulls the latest Docker image:

./bigeye-agent source upgrade
./bigeye-agent lineage upgrade
./bigeye-agent datahealth upgrade
./bigeye-agent sds upgrade
./bigeye-agent crosssource upgrade
./bigeye-agent external-monitors upgrade

Updated about 6 hours ago