Microsoft Purview Lineage Integration

The Bigeye Data Health Agent can replicate the lineage Bigeye knows about into Microsoft Purview, so your warehouse → transformation → BI lineage shows up natively on your Purview assets. This guide walks through everything required to configure and run it.

Status: This integration is in preview. The steps below are accurate for the current release and target unified Microsoft Purview.

1. What the integration does

• Direction: one-way, Bigeye → Purview. It never writes back to Bigeye.
• What it writes: Purview built-in direct_lineage_dataset_dataset relationships between
  your assets:
  • Column-level lineage (a columnMapping) wherever Bigeye has column-level lineage
    (e.g. table → table, table → BI report).
  • Table-level lineage (no columnMapping) where Bigeye only has object-level lineage
    (e.g. BI report → BI dashboard).
• Un-scanned intermediate nodes: if Bigeye knows about one or more ETL nodes (e.g. a
  replication task) that Purview has not scanned between two mapped assets, the agent creates a
  lightweight placeholder dataset for each hop under the qualified-name prefix
  bigeye://lineage/... so the whole source → target chain stays connected.
• Idempotent reconcile: each run creates new edges, updates changed ones in place, and
  removes edges Bigeye no longer reports. Re-running is safe. Bigeye owns the lineage it
  writes — do not hand-edit Bigeye-authored edges in Purview.

How matching works (why you only tag a few assets)

You do not tag every table. The agent resolves assets by searching Purview:

• Tables/views are matched per (source, schema): the agent finds your source-level
  asset (tagged with a Bigeye Source ID), then searches for the schema and its tables
  underneath it. Tag the source asset once; every table under it is matched automatically.
• BI objects (reports, dashboards) are matched through their parent workspace: tag
  the BI workspace asset (e.g. a fabric_workspace) with a Bigeye DataNode ID, and the
  reports/dashboards under it are matched automatically.

The agent never constructs qualified names — it only references assets returned by search.

2. Prerequisites

On the Bigeye side

• A Bigeye API key (or username/password) with access to the workspace you want to sync.
• The workspace ID of the workspace you want to sync. The sync only covers the sources and
  BI assets you map in Purview (see the callout above) — not every source in the workspace.
• Lineage must already exist in Bigeye. The agent replicates Bigeye's lineage graph; if
  Bigeye has no lineage for a source, nothing is written for it.

On the Purview side

• A unified Microsoft Purview account.
• Permission to create an Azure AD app registration and to assign Purview collection roles.
• Your warehouses and BI tools already scanned into Purview (the agent attaches lineage
  to assets Purview has discovered).
• Purview's own lineage generation disabled where possible for the sources/BI Bigeye will
  manage (see the callout at the top) — Bigeye is the source of truth for their lineage. Some
  source types don't allow it to be disabled; see the callout for what to expect.

Network egress (the agent must reach, over HTTPS/443):

• login.microsoftonline.com — Azure AD token endpoint
• Your Purview data-plane endpoint — https://api.purview-service.microsoft.com
• Your Bigeye instance — e.g. https://app.bigeye.com
• Your secret store, if you load the credential from one (AWS/Azure/GCP/HashiCorp)

Host resources

The pre-install checklist sets the agent's baseline at 1 CPU / 2 GB (4 CPU / 16 GB if it shares a host with the Bigeye agent). sync-lineage is the most memory-hungry command the agent runs. Unlike the other sync commands it holds the whole lineage graph in memory for the duration of the run, and the footprint scales with the number of lineage nodes rather than the number of tables. So provision 4 GB if you are syncing lineage for a sources at larger scales; e.g. >= 15,000 tables.

3. Step 1 — Create an Azure AD app registration (OAuth)

The agent authenticates to Purview as a service principal using the OAuth2 client-credentials
flow.

1. In the Azure portal → Microsoft Entra ID (Azure AD) → App registrations → New
   registration. Give it a name (e.g. bigeye-purview-lineage) and register it.
2. Note the Application (client) ID and the Directory (tenant) ID.
3. Under Certificates & secrets → New client secret, create a secret and copy its
   value immediately (you cannot retrieve it later). Note its expiry — you must rotate the
   credential before it expires.
4. Grant the service principal a Purview role. This is the step most often missed. In the
   Purview governance portal, open Data Map → Collections (or Roles), select the
   collection(s) that contain the assets you want lineage on (use the root collection to
   cover everything), and add the app registration as a Data Curator.
   • Data Curator is required because the agent must read assets and managed attributes
     and write lineage relationships (and placeholder datasets).
   • The agent requests the token scope https://purview.azure.net/.default automatically;
     you only supply the tenant/client/secret.

4. Step 2 — Create the "Bigeye Object IDs" managed attributes

The agent finds your Purview assets by reading custom managed attributes (also called
custom/business metadata) that you set on them. Create one attribute group with two attributes.

Names and types must match exactly — the agent looks them up by these literal names.

1. In Purview, open Unified Catalog → Catalog management → Custom metadata click on the Data asset attributes tab.

2. Create a new attribute group named exactly:

   Bigeye Object IDs

3. Add two attributes to that group:

   Attribute name	Type	Associated asset types
   Bigeye Source ID	Integer	Your source/server asset types (e.g. databricks_metastore, azure_sql_server, Snowflake/Oracle server types)
   Bigeye DataNode ID	Integer	Your BI workspace asset type (e.g. fabric_workspace)

   • The type must be Integer — the agent matches on the numeric value.
   • "Associated asset types" must include the asset types you intend to tag in Step 3,
     otherwise the attribute can't be set on (or read from) those assets.
   • If you have multiple asset types, you can simply use Asset under Atlas Core

5. Step 3 — Map Bigeye IDs onto Purview assets

For each source and BI workspace you want lineage for, set the managed attribute on the
corresponding Purview asset.

Sources (warehouses):

1. Find the Bigeye Source ID — the numeric ID of the warehouse/source in Bigeye. You can
   retrieve it from the Bigeye API (GET {bigeye_base_url}/api/v1/sources) or from the
   source's page URL in the Bigeye app.

2. In Purview, open the source/server-level asset for that warehouse (not an individual
   table) and set Bigeye Object IDs → Bigeye Source ID to that number.
   

   

BI workspaces:

1. Find the Bigeye DataNode ID for the BI tool from the Lineage section of Bigeye —
   not from the lineage tab of a linked table. Navigate to the BI tool's node in the
   Lineage view; the node ID appears in the page URL.

   

2. In Purview, open the BI workspace asset (e.g. the fabric_workspace) and set
   Bigeye Object IDs → Bigeye DataNode ID to that number.

BI lineage requires a mapped warehouse source. Syncing BI tools on their own isn't
supported yet — BI lineage is replicated only where it connects to a warehouse source you've
mapped. Be sure to map the relevant warehouse source(s) as well.

You only need to tag the source-level and workspace-level assets — tables, reports, and
dashboards under them are matched automatically (see How matching works above).

Mapping defines the sync's scope. The integration only creates lineage for the sources
and BI workspaces you map here. Sources you don't map are ignored, and lineage that only
routes through an unmapped source (for example, a table in an unmapped warehouse sitting
between two mapped tables) is dropped rather than bridged. To narrow a sync, map fewer
sources; to widen it, map more.

6. Step 4 — Build the credential

The credential is a single object you can provide as a JSON file or store in a
secret manager. Fields:

purview_tenant_id, purview_client_id, and purview_client_secret each support
${ENV_VAR} substitution — e.g. set purview_client_secret: ${PURVIEW_CLIENT_SECRET} and
provide the value via an environment variable instead of in the file.

Example (purview-config.json):

{
  "bigeye_base_url": "https://app.bigeye.com",
  "bigeye_api_key": "<your-bigeye-api-key>",
  "bigeye_workspace_id": 123,
  "purview_tenant_id": "00000000-0000-0000-0000-000000000000",
  "purview_client_id": "11111111-1111-1111-1111-111111111111",
  "purview_client_secret": "someclientsecret"
}

7. Step 5 — Run sync-lineage

Download datahealth-integration-docker-compose.yaml as docker-compose.yaml and create the necessary directory with credential file above. The commands below show how to run the sync via Docker compose

wget https://bigeye-public-web.s3.amazonaws.com/datahealth-integration-docker-compose.yaml --output-document=docker-compose.yaml

mkdir bigeye_datahealth_config
touch bigeye_datahealth_config/purview-config.json

From a local file:

docker compose run --rm datahealth-integration sync-lineage --file-name purview-config.json

From AWS Secrets Manager:

docker compose run --rm datahealth-integration sync-lineage --aws-secret-name my/purview/cred --aws-region us-west-2

From Azure Key Vault:

docker compose run --rm datahealth-integration sync-lineage \
  --azure-secret-name my-purview-cred \
  --azure-vault-url https://my-vault.vault.azure.net/

From GCP Secret Manager:

docker compose run --rm datahealth-integration sync-lineage --gcp-secret-name my-purview-cred --gcp-project-id my-project

From HashiCorp Vault:

docker compose run --rm datahealth-integration sync-lineage \
  --hashicorp-vault-url https://vault.example.com \
  --hashicorp-secret-path secret/data/bigeye/purview \
  --hashicorp-token <token>

Tuning the lineage crawl (optional):

Reading the lineage graph is the longest part of the sync — one request per lineage node, and a
large source has tens of thousands. Two optional settings control it, each available as a flag or
an environment variable.

docker compose run --rm datahealth-integration sync-lineage \
  --file-name purview-config.json \
  --lineage-crawl-workers 8

A successful run logs a summary such as:

Skipping 40 of 9,000 table(s) that have no lineage node yet; there is nothing to crawl for them until indexing creates one.
Retrieving lineage from 3 sources...
Lineage crawl wave 1: 9,000 node(s) to fetch.
Lineage crawl complete: 13000 nodes built over 3 waves and 13000 fetches in 957.9s (seeds dropped: 0, not found: 0, unreadable: 0, peak RSS: 458MB)
Resolved 3 source(s) and 1 BI object(s) to Purview assets.
Matched 12942 node(s) to Purview entities.
Reconciling 12942 Bigeye lineage edge(s) in Purview...
Lineage reconcile: 12942 created, 0 updated, 0 deleted.

8. Scheduling

Run sync-lineage on a batch schedule (hourly/daily is typical) using whatever scheduler runs
the container in your environment — for example a Kubernetes CronJob, an ECS scheduled task,
or cron invoking the Docker image:

# Example cron entry (daily at 02:00)
0 2 * * * docker run --rm \
  -e PURVIEW_CLIENT_SECRET=*** \
  -v /etc/bigeye/purview-config.json:/config.json \
  <bigeye-datahealth-agent-image> sync-lineage --file-name /config.json

9. Verify it worked

1. Check the run log for the summary above. Resolved N source(s) / M BI object(s) confirms
   your managed-attribute tags were found.
2. In Purview, open a table under a tagged source (or a tagged BI report) and view its
   Lineage tab — you should see the Bigeye-derived upstream/downstream edges, with
   column-level mapping where Bigeye has it.

10. Troubleshooting

11. Limitations & behavior to expect

• One-directional — Bigeye → Purview.
• Requires existing Bigeye lineage and scanned Purview assets to connect to.
• Reconcile deletes Bigeye-authored edges that are no longer reported, and prunes stale
  bigeye://lineage/... placeholder nodes. Don't manually edit Bigeye-authored lineage in
  Purview — it will be overwritten on the next run.
• An incomplete read fails the run instead of deleting lineage. Because reconcile removes edges
  Bigeye no longer reports, a crawl that silently missed nodes would delete real lineage. If more
  than --max-unreadable-nodes (default 0) nodes can't be read after retries, the run fails
  having written nothing. Transient failures are retried automatically and rarely reach this point.
• Newly added tables sync on a later run — a table has no lineage node until Bigeye indexes it.
  Those tables are counted in the log and skipped until then.
• Schema/object names that differ between Bigeye and Purview cause those edges to be skipped
  (logged) rather than mismatched.
• Scoped to what you map — only the sources and BI assets mapped in Purview are synced;
  unmapped sources, and lineage that only routes through an unmapped warehouse table, are
  excluded (the dropped edges are logged).
• BI-only sync isn't supported yet. The sync seeds from mapped warehouse sources, so a
  BI workspace's lineage is replicated only where it connects to a mapped warehouse source. You
  must map at least one warehouse source; mapping only BI workspaces won't sync any lineage.
• ETL chains become placeholder datasets — lineage flowing through one or more ETL nodes
  Purview hasn't scanned is reconstructed as a chain of bigeye://lineage/... placeholder
  datasets, one per hop, so the source → target path stays connected.
• Where Purview auto-lineage can't be disabled, Bigeye's lineage may coexist with
  Purview's, or Bigeye may fail to create an edge where lineage already exists (logged, run
  continues). See the source-of-truth callout at the top.