Kubernetes Deployment

Some Bigeye agents can be deployed on Kubernetes instead of Docker. The Agent CLI generates Kubernetes YAML charts and provides step-by-step kubectl commands for deployment.

Supported Agents

For agents not yet supported on Kubernetes, contact Bigeye support.

Initial Setup

During ./bigeye-agent install, select Kubernetes as your deployment target. This sets deploy_on_kubernetes: true in your configuration and prompts for a Kubernetes namespace (default: bigeye).

The Agent CLI still needs to be run locally to generate configuration files. The generated YAML charts are then applied to your cluster.

Data Source Agent on Kubernetes

After running ./bigeye-agent source configure, a deployment chart is generated at bigeye-agent-deployment.yaml.

Prerequisites

# Create the namespace
kubectl create namespace bigeye

# Create configmap from agent config directory
kubectl create configmap bigeye-agent-config \
  --namespace bigeye \
  --from-file=agent_config/

Deploy

kubectl apply -f bigeye-agent-deployment.yaml

Lineage Plus Agent on Kubernetes

Partial Support: The Kubernetes deployment does not support scenarios where custom JARs are required for lineage collection, or configurations exceeding the 1 MiB Kubernetes ConfigMap size limit.

Prerequisites

1. Install the Lineage Plus agent and add connectors using the CLI:

./bigeye-agent install       # Select LINEAGE_PLUS
./bigeye-agent add-connector  # Add your connectors

This creates the lineage_config/ directory with all necessary files.

2. Run a connector to output the necessary Kubernetes charts (Example below shows Snowflake):

# Run snowflake
./bigeye-agent lineage run -c snowflake

📘

Running a connector

When "running" a connector on K8s, the CLI just outputs all the necessary charts to run a lineage job on a cluster. Additionally, it prints out the steps required for creating the config and then running the connector.

Configure

1. Follow the steps printed out by the CLI to deploy the connector configuration.

Deploy

1. Follow the steps printed out by the CLI to deploy the job to run the connector

DataHealth Agent on Kubernetes

The CLI generates a CronJob chart at bigeye-datahealth-cronjob.yaml with a default schedule of daily at 00:00 UTC. Edit the chart to customize the schedule before applying.

# Run datahealth for a workspace
./bigeye-agent datahealth run -wid 4

# Follow the instructions output by the CLI