Lineage API

This is a short walkthrough on how to use the Bigeye API to create a few parent-child relationships between tables. This is called Table Lineage in Bigeye. Using those relationships, we will then issue a mute API call that will mute all metrics for not only a single table, but all of the dependent table's metrics as well.

This can be useful for ELT pipelines where data flows from Table A -> Table B -> Table C. If the data in Table A has a data quality problem, it's fairly common to want to silence alerts for all of that table's dependent tables (B & C) as well.

Pre-steps

The API Reference is a useful resource as it catalogs all of the operations that our API supports.

Also the API User guide is a good for getting started with making API calls to Bigeye.

Getting a Table's Lineage

The Lineage API operates on data node ids, so the first step to retrieve a table's lineage is to find the data node id, by querying the dataset API with the id of the table. The table id can be found on the page for the table in Bigeye, which should look something like https://staging.bigeye.com/preview/catalog/table//columns. Then, you want to make the following call, which will return the data node id if it exists for a table. If it does not exist, create the data node for a table first.

GET /dataset/<table_id>

Response:
{
    ...
    dataNodeId: <data_node_id>
    ...
}

Creating a Data Node for a Table

Data nodes are separate from tables themselves, even though they reference the same table entity. Thus, in order to create a relationship for a table that doesn't have a data node, you first need to create a data node for a table.

POST /api/v1/lineage/nodes

body:
{
    nodeType: "DATA_NODE_TYPE_TABLE",
    nodeEntityId: <table_id>
}

Response:
{
    id: <data_node_id>,
    companyId: <company_id>,
    nodeType: <data_node_type>,
    nodeEntityId: <data_node_entity_id>, // this is the table_id if this is a table data node type, otherwise it has an id that does not correspond to anything that exists currently
    nodeName: <data_node_name>, // this is the table name if this is a table data node type, otherwise it does not correspond to anything that exists currently
    nodeContainerName: <data_node_container_name>, // this is the table schema name if this is a table data node type, otherwise it does not correspond to anything that exists currently
    createdAt: <created_at_epoch_seconds>,
    updatedAt: <updated_at_epoch_seconds>
}

List a Table's Lineage

This should be empty since we haven't set up any table lineage yet, but it is good to check before we begin. This GET should return 2 empty lists of the relationships that are upstream or downstream of that node.

GET /api/v1/lineage/nodes/{data_node_id}/relationships

Response:
{
    "upstream": []
    "downstream": []
}

Create Table Relationship

In this example, we'll create a relationship:
Table A -> Table B

where the data node ids are:
Table A = 1
Table B = 2

POST /api/v1/lineage/relationships

body:
{
    "upstreamDataNodeId": 1,
    "downstreamDataNodeId": 2
}

Mute Metrics for All 3 Tables in 1 Call

Issuing a mute API call to mute metrics on Table A with includeDecendants=true will cascade the mute to all metrics on Table B and Table C.

POST /api/v1/tables/992/snooze

body:
{
  "minutes": 60,
  "includeDescendants": true
}

Unmute Metrics for All 3 Tables in 1 Call

Unmuting works in a similar fashion.

POST /api/v1/tables/992/unsnooze

body:
{
  "includeDescendants": true
}