Guides

Connect Databricks to Agent Trust Hub

Agent Trust Hub brings your Databricks Genie activity into Bigeye. Once connected, Bigeye discovers each Genie Space as an AI agent, records the conversations and turns that run against it, and attributes the columns and tables each turn read—so you can see, govern, and alert on what your AI agents are doing with your data.

This guide covers the extra setup Agent Trust Hub needs on top of a standard Databricks connection. The sync is read-only: Bigeye never writes to your Databricks workspace and never runs SQL there—it only reads Genie metadata through the Databricks REST API.

📘

Connect the source first

Agent Trust Hub reuses the Databricks source you connect in Connect Databricks. Set up the service principal and connection there first, then return here to grant the Genie permissions and enable Agent Trust Hub.

1. What Gets Synced

Agent Trust Hub maps Databricks Genie objects to Bigeye Agent Trust Hub objects:

  • Genie Space → an AI agent
  • Conversation → a conversation
  • Message (one question + answer turn) → a span, where input is the user's question and output is Genie's answer
  • SQL a turn executed → data access: the columns and tables the turn read

2. How Authentication Works

Agent Trust Hub reuses the connection details of your Databricks source in Bigeye. It calls the Databricks Genie Conversations REST API on your workspace host:

GET https://<workspace-host>/api/2.0/genie/spaces
GET https://<workspace-host>/api/2.0/genie/spaces/{space_id}
GET https://<workspace-host>/api/2.0/genie/spaces/{space_id}/conversations
GET https://<workspace-host>/api/2.0/genie/spaces/{space_id}/conversations/{conversation_id}/messages

Each request authenticates with the service principal's access token as a bearer credential:

Authorization: Bearer <access token>
🚧

Access Token authentication only

Because Genie is called with a bearer token, the Databricks source must be connected using Access Token authentication. OAuth M2M works for Bigeye's data-quality features but is not supported for Agent Trust Hub today—the OAuth client secret is not a bearer token and Genie will reject it.

3. Prerequisites

  • A Databricks workspace with Genie enabled and at least one Genie Space.
  • The Databricks service principal from Connect Databricks, with the additional Genie permissions in Section 4.
  • An access token for that service principal (a bearer token). Agent Trust Hub does not support OAuth M2M.
  • A Bigeye workspace with Agent Trust Hub enabled (see Section 6).
  • (Optional, for column-level attribution) the Databricks catalog, schemas, and tables ingested into Bigeye as part of the same source.

4. Grant Genie Permissions to the Service Principal

Grant these to the same service principal you created in Connect Databricks. They are in addition to the Unity Catalog or Hive Metastore grants that guide already covers. Grant the least privilege that covers the Spaces you want to sync.

4.1 Workspace Entitlements

Set these under Settings → Identity and access → Service principals → select the service principal → Entitlements.

  • Workspace access (workspace-access): sign in to the workspace and call its REST API.
  • Databricks SQL access (databricks-sql-access): required to use Genie and the Genie Conversations API.

4.2 Genie Space Access

Set these on each Space via its Share / permissions dialog.

  • CAN VIEW (minimum): discover the Space and read its metadata. Bigeye only ever sees Spaces the service principal can see.
  • CAN MANAGE (recommended): read the full conversation history across all users in the Space. With only CAN VIEW or CAN RUN, the API may return just the service principal's own conversations, which undercuts governance. Grant CAN MANAGE (or make the service principal a Space owner) to capture everyone's activity.

4.3 SQL Warehouse

  • CAN USE on the SQL warehouse backing the Space: Genie runs on a SQL warehouse, and Bigeye's data-quality connection uses it over the HTTP path.

4.4 Unity Catalog

Required only for column-level data-access attribution, so Bigeye can ingest the catalog objects the agents query. Grant on the catalogs, schemas, and tables you want resolved:

  • USE CATALOG on each catalog: traverse into the catalog.
  • USE SCHEMA on each schema: traverse into the schema.
  • SELECT on the tables/views: read table metadata (and sample data for profiling).

If you skip the Unity Catalog grants, Agent Trust Hub still works—it records the tables each turn sourced instead of individual columns (see Section 8).

📘

Least privilege

Account admin, workspace admin, and Unity Catalog metastore admin are not required. Prefer a dedicated service principal over a personal user token for a stable, auditable, rotatable connection.

5. Connect Databricks with Access Token Authentication

Follow Connect Databricks to add the source, choosing Access Token Authentication and supplying the service principal's access token.

The Host and access token are what Agent Trust Hub uses to reach Genie. The HTTP path is used for Bigeye's SQL-based data-quality features and is not required by the Genie sync—but connecting the source's catalog is what enables column-level data access attribution (see Section 8).

6. Enable Agent Trust Hub

Agent Trust Hub is enabled per Bigeye workspace. If you don't yet see AI agents after connecting Databricks, ask your Bigeye account team to enable Agent Trust Hub for your workspace. Once enabled, any Databricks source in that workspace is picked up automatically.

7. Sync and View Results

  • Automatic: Agent Trust Hub syncs your Databricks Genie activity on a nightly schedule. New conversations and turns are added on each run; existing ones are not duplicated.
  • On demand: need results sooner? Ask your Bigeye account team to trigger an immediate re-sync for the source.

After a sync you'll see, per Databricks source:

  • One AI agent for each Genie Space.
  • Each Space's conversations, and within them each turn (the question asked and the answer Genie returned).
  • The data each turn accessed—the tables, and where the catalog is ingested, the specific columns.

8. Data Access Attribution

For every turn that ran SQL, Bigeye parses the query to determine which columns and tables were read and attributes that access to the conversation.

  • If the Databricks catalog/schema/table is ingested into Bigeye, access resolves to specific columns.
  • If the source objects are not ingested (or the SQL can't be parsed), Bigeye falls back to recording the tables Genie reported sourcing, so the access is still captured.

To get column-level attribution, make sure the relevant Unity Catalog catalogs and schemas are included when you connect or refresh the Databricks source.

9. Data Handling and Privacy

  • The sync is read-only over the Databricks REST API. Bigeye does not modify Genie Spaces or run SQL in your workspace.
  • If your Bigeye workspace is configured to block all row-level access (data-restricted mode), Agent Trust Hub does not store raw prompts or responses—only conversation and turn identifiers plus the data-access attribution are recorded.
  • The service principal's access token is stored encrypted, the same way as every other Bigeye source credential.

10. Troubleshooting

No AI agents appear after connecting.

  • Confirm Agent Trust Hub is enabled for your workspace (Section 6).
  • Confirm the source uses Access Token authentication, not OAuth M2M.
  • Confirm the service principal can see the Genie Spaces (CAN VIEW).

Only tables show up under data access, not columns.

  • Column-level resolution requires the Databricks catalog/schema/tables to be ingested into Bigeye. Add them to the source, then re-sync.

401/403 errors, or agents stop updating.

  • The access token has expired or lost permissions. Generate a new token for the service principal and update it on the source (Catalog → your source → Edit → Access token).

Some Spaces or conversations are missing.

  • Bigeye only syncs Spaces the service principal can view—grant at least CAN VIEW on the Spaces you expect to see.
  • If Spaces appear but only some conversations sync, the service principal likely has CAN VIEW or CAN RUN but not CAN MANAGE. Grant CAN MANAGE on the Space to capture the full conversation history from all users.

11. Limitations

  • Authentication: Access Token only. OAuth M2M is not yet supported for Agent Trust Hub.
  • Volume: each sync pulls the most recent Spaces and conversations up to a configured limit per source.
  • Backfill: new turns added to a conversation are backfilled on later syncs; a conversation that has already been synced is not re-created.

Updated 21 days ago

Did this page help you?