Lineage API
This is a short walkthrough on how to use the Bigeye API to create a few parent-child relationships between tables. This is called Table Lineage in Bigeye. Using those relationships, we will then issue a mute API call that will mute all metrics for not only a single table, but all of the dependent table's metrics as well.
This can be useful for ELT pipelines where data flows from Table A -> Table B -> Table C. If the data in Table A has a data quality problem, it's fairly common to want to silence alerts for all of that table's dependent tables (B & C) as well.
Pre-steps
The API Reference is a useful resource as it catalogs all of the operations that our API supports.
Also the API User guide is a good for getting started with making API calls to Bigeye.
Getting a Table's Lineage
The Lineage API operates on data node ids, so the first step to retrieve a table's lineage is to find the data node id, by querying the dataset API with the id of the table. The table id can be found on the page for the table in Bigeye, which should look something like https://staging.bigeye.com/preview/catalog/table//columns. Then, you want to make the following call, which will return the data node id if it exists for a table. If it does not exist, create the data node for a table first.
GET /dataset/<table_id>
Response:
{
...
dataNodeId: <data_node_id>
...
}
Creating a Data Node for a Table
Data nodes are separate from tables themselves, even though they reference the same table entity. Thus, in order to create a relationship for a table that doesn't have a data node, you first need to create a data node for a table.
POST /api/v1/lineage/nodes
body:
{
nodeType: "DATA_NODE_TYPE_TABLE",
nodeEntityId: <table_id>
}
Response:
{
id: <data_node_id>,
companyId: <company_id>,
nodeType: <data_node_type>,
nodeEntityId: <data_node_entity_id>, // this is the table_id if this is a table data node type, otherwise it has an id that does not correspond to anything that exists currently
nodeName: <data_node_name>, // this is the table name if this is a table data node type, otherwise it does not correspond to anything that exists currently
nodeContainerName: <data_node_container_name>, // this is the table schema name if this is a table data node type, otherwise it does not correspond to anything that exists currently
createdAt: <created_at_epoch_seconds>,
updatedAt: <updated_at_epoch_seconds>
}
List a Table's Lineage
This should be empty since we haven't set up any table lineage yet, but it is good to check before we begin. This GET should return 2 empty lists of the relationships that are upstream or downstream of that node.
GET /api/v1/lineage/nodes/{data_node_id}/relationships
Response:
{
"upstream": []
"downstream": []
}
Create Table Relationship
In this example, we'll create a relationship:
Table A -> Table B
where the data node ids are:
Table A = 1
Table B = 2
POST /api/v1/lineage/relationships
body:
{
"upstreamDataNodeId": 1,
"downstreamDataNodeId": 2
}
Mute Metrics for All 3 Tables in 1 Call
Issuing a mute API call to mute metrics on Table A with includeDecendants=true will cascade the mute to all metrics on Table B and Table C.
POST /api/v1/tables/992/snooze
body:
{
"minutes": 60,
"includeDescendants": true
}
Unmute Metrics for All 3 Tables in 1 Call
Unmuting works in a similar fashion.
POST /api/v1/tables/992/unsnooze
body:
{
"includeDescendants": true
}
Updated 12 months ago