Available Metrics

This is an overview of all metrics available in Bigeye. Bigeye categorizes metrics into data quality dimensions to better help you summarize and track progress over time.

Pipeline Reliability

Pipeline reliability metrics detect whether tables are updating on time and with the expected volume of data.

Metric NameAPI NameDescription
FreshnessFRESHNESSFor a given table update, the number of hours since the previous table update (INSERT, COPY, MERGE, CREATE, etc). Available on Snowflake, BigQuery, and Redshift sources, excluding database views.
VolumeVOLUMEFor a given table update, the number of rows inserted or upserted to the table (INSERT, COPY, MERGE, CREATE TABLE AS SELECT, etc). Available on Snowflake, BigQuery, and Redshift sources, excluding database views.
Freshness (Data) FRESHNESS_DATAThis is similar to Freshness but based on data, not metadata. This is available on source types other than snowflake, bigquery, and redshift, and also available for all views. It mimics the behavior of Freshness by recording the time of the actual loads, not by measuring the time since a load.
Volume (Data)VOLUME_DATAThis is similar to Volume but based on data, not metadata. This is available on source types other than snowflake, bigquery, and redshift, and also available for all views. It mimics the behavior of Volume.
Hours since latest valueHOURS_SINCE_MAX_TIMESTAMP or HOURS_SINCE_MAX_DATEApplicable to DATE_LIKE or TIMESTAMP_LIKE columns. The difference between the metric run time and the maximum value of the timestamp column, in hours. Hours since latest value is suggested as a basic autometric on all date
and timestamp columns.
Row Count (#)COUNT_ROWSThe total number of rows in a table. It is suggested as a basic autometric once per table.
Read queriesCOUNT_READ_QUERIESThe number of SELECT queries issued on a table in the past 24 hours. It is suggested as a basic autometric once per table.

Uniqueness

Uniqueness metrics detect when schema and data constraints are breached.

Metric NameAPI NameColumn TypeDescription
Distinct (#)COUNT_DISTINCTANYThe count of distinct elements in the column. This metric should be used when you expect a fixed number of value options. It is suggested as an autometric if Bigeye detects 50 or fewer values during profiling.
Duplicates (#)COUNT_DUPLICATESANYThe count of rows with the same value for a particular column. It is suggested as an autometric if Bigeye detects 10 or fewer duplicates during profiling.

Completeness

Completeness metrics detect when there are missing values in datasets.

Column NameAPI NameColumn TypeDescription
Null (#)COUNT_NULLANYThe count of rows with a null value in the column.
Not Null (#)COUNT_NOT_NULLANYThe count of rows with a non-null value in the column.
Null (%)PERCENT_NULLANYThe percentage of rows with a null value in the column. This metric is suggested as a basic autometric on all column types.
Not Null (%)PERCENT_NOT_NULLANYThe percentage of rows with a non-null value in the column.
Empty string (#)COUNT_EMPTY_STRINGSTRINGThe count of rows with a 0-length string (i.e. "") as the value for the column.
Empty string(%)PERCENT_EMPTY_STRINGSTRINGThe percent of rows with a 0-length string. It is suggested as an autometric if Bigeye detects >= 50% of values that match during profiling or if the column is an ID column.
NaN (#)COUNT_NANNUMERICThe count of rows where the column value is NaN. This metric will only be available for source types where NaN is a valid value for a column.
NaN (%)PERCENT_NANNUMERICThe percentage of rows where the column value is NaN. This metric will only be available for source types where NaN is a valid value for a column. It is suggested as an autometric if Bigeye detects >= 50% of values during our profiling.

Distributions

Distribution metrics detect changes in the numeric distribution of values, including outliers, variance, skew and more.

Column NameAPI NameColumn TypeDescription
MinMINNUMERICThe minimum value of the column. It is suggested as a basic autometric for all numeric columns.
MaxMAXNUMERICThe maximum value of the column. It is suggested as a basic autometric for all numeric columns.
AverageAVERAGENUMERICThe mean value of the column. It is always suggested as a basic autometric for numeric columns, except for ID columns.
VarianceVARIANCENUMERICThe statistical variance of the column. The variance is used to track the spread of numbers beyond the average. It is always suggested as a basic autometric for numeric columns, except for ID columns.
SkewSKEWNUMERICThe statistical skew of the column. The skew is used to determine how evenly the values are distributed about the mean. A negative skew means that there is a larger tail below the mean, while a positive skew indicates a larger tail above the mean.
KurtosisKURTOSISNUMERICThe statistical kurtosis of the column. The kurtosis determines how much of a tail datasets have. The value displayed is actually the excess kurtosis, where 3 is subtracted from the kurtosis value, so a normal distribution would end up with a metric value of 0.
Geometric meanGEOMETRIC_MEANNUMERICThe geometric mean of the column.
Harmonic meanHARMONIC_MEANNUMERICThe harmonic mean of the column.
MedianMEDIANNUMERICThe median of the column. The median is computed as the 50th percentile, and will only return a value that is in the dataset. It is not valid for the MySQL source type. It is always suggested as a basic autometric for numeric columns, except for ID columns.
PercentilePERCENTILENUMERICThe statistical percentile of the column. This metric takes a parameter to determine what percentile should be used. Values less than one as well as less than 100 are accepted, where a 90th percentile can be expressed as either 0.9 or 90. Bigeye computes percentile using the discrete percentile, where only existing values will be returned, except for the Presto or AWS Athena source types. Not valid for MySQL source type.
SumSUMNUMERICThe sum of all values in the column. It is always suggested as a basic autometric for numeric columns, except for ID columns.
False (#)COUNT_FALSEBOOLEANThe count of rows where the column contains the boolean value of false.
False (%)PERCENT_FALSEBOOLEANThe percentage of rows where the column contains the boolean value of false. It is suggested as a basic autometric on all boolean columns.
True (#)COUNT_TRUEBOOLEANThe count of rows where the column contains the boolean value of true.
True (%)PERCENT_TRUEBOOLEANThe percentage of rows where the column contains the boolean value of true. It is suggested as a basic autometric on all boolean columns.

Validity

Validity metrics detect whether data is formatted correctly and represents a valid value. Bigeye offers validity metrics across a number of categories shown below.

String formats

Column NameAPI NameColumn TypeDescription
String Length MaxSTRING_LENGTH_MAXSTRINGThe maximum value of the column's length. Not valid for Oracle source type. It is suggested as a basic autometric for all string columns.
String Length MinSTRING_LENGTH_MINSTRINGThe minimum value of the column's length. Not valid for Oracle source type. It is suggested as a basic autometric for all string columns.
String Length AverageSTRING_LENGTH_AVERAGESTRINGThe average value of the column's length. Not valid for Oracle source type. It is suggested as a basic autometric for all string columns.

Identification formats

You can run debug query on alerting metrics under Identification formats group.

Metric NameAPI NameColumn TypeDescription
UUID (#)COUNT_UUIDSTRINGThe number of rows where the column matches the UUID format with hyphens (i.e. 123e4567-e89b-12d3-a456-426614174000). The comparison is case insensitive.
UUID(%)PERCENT_UUIDSTRINGThe percentage of rows where the column matches the UUID format with hyphens (i.e. 123e4567-e89b-12d3-a456-426614174000). The comparison is case insensitive. It is suggested as an autometric if Bigeye detects a match greater than 50% during profiling.
Perm ID (#)COUNT_PERM_IDSTRINGThe number of rows where the column matches an approximation of a valid Perm ID format (currently 1-<1-15 digits>)
Perm ID (%)PERCENT_PERM_IDSTRINGThe percentage of rows where the column matches an approximation of a valid Perm ID format (currently 1-<1-15 digits>). It is suggested as an autometric if Bigeye detects a match greater than 50% during profiling.
SSN (#)COUNT_SSNSTRINGThe number of rows where the column matches an approximation of a valid Social Security number, with or without hyphens.
SSN (%)PERCENT_SSNSTRINGThe percentage of rows where the column matches an approximation of a valid Social Security number, with or without hyphens. It is suggested as an autometric if Bigeye detects a match greater than 50% during profiling.

Contact Information

You can run debug query on alerting metrics under Contact Information group.

Metric NameAPI NameColumn TypeDescription
USA Phone number (#)COUNT_USA_PHONESTRINGThe number of rows where the column matches USA phone number format, with or without country code, parethenses, or hyphens. Examples: 1 (401) 555 6789, 405-555-6789, 5556789
USA Phone number (%)PERCENT_USA_PHONESTRINGThe percentage of rows where the column matches USA phone number format, with or without country code, parethenses, or hyphens. It is suggested as an autometric if Bigeye detects a match >= 50% during profiling.
USA State Code (#)COUNT_USA_STATE_CODESTRINGThe number of rows where the column matches the state codes of the 50 US states. The comparison is case insensitive.
USA State Code (%)PERCENT_USA_STATE_CODESTRINGThe percentage of rows where the column matches the state codes of the 50 US states. The comparison is case insensitive. It is suggested as an autometric if Bigeye detects a match >= 50% during profiling.
USA ZIP Code (#)COUNT_USA_ZIP_CODESTRINGThe number of rows where the column matches the ZIP code (12345) or the ZIP+4 (12345-1234) format.
USA ZIP Code (%)PERCENT_USA_ZIP_CODESTRINGThe percentage of rows where the column matches the ZIP code (12345) or the ZIP+4 (12345-1234) format. It is suggested as an autometric if Bigeye detects a match >= 50% during profiling.
Email (#)COUNT_EMAILSTRINGThe number of rows where the column matches an approximation of a valid email address. Some source types are slightly more precise than others due regex compatibility, but any source will recognize a large majority of emails.
Email (%)PERCENT_EMAILSTRINGThe percentage of rows where the column matches an approximation of a valid email address. Some source types are slightly more precise than others due regex compatibility, but any source will recognize a large majority of emails. It is suggested as an autometric if Bigeye detects a match >= 50% during profiling.

Financial

You can run debug query on alerting metrics under Financial group.

Metric NameAPI NameColumn TypeDescription
SEDOL (#)COUNT_SEDOLSTRINGThe number of rows where the column matches the Stock Exchange Daily Official List format. Bigeye does not verify the checksum.
SEDOL (%)PERCENT_SEDOLSTRINGThe percentage of rows where the column matches the Stock Exchange Daily Official List format. We do not verify the checksum. It is suggested as an autometric if Bigeye detects a match greater than 50% during profiling.
CUSIP (#)COUNT_CUSIPSTRINGThe number of rows where the column matches the Committee on Uniform Securities Identification Procedures format. Bigeye does not verify the check digits.
CUSIP (%)PERCENT_CUSIPSTRINGThe number of rows where the column matches the Committee on Uniform Securities Identification Procedures format. Bigeye does not verify the check digits. It is suggested as an autometric if Bigeye detects a match greater than 50% during profiling.
LEI (#)COUNT_LEISTRINGThe number of rows where the column matches the Legal Entity Identifier format. Bigeye does not verify the check digits.
LEI (%)PERCENT_LEISTRINGThe percentage of rows where the column matches the Legal Entity Identifier format. Bigeye does not verify the check digits. It is suggested as an autometric if Bigeye detects a match greater than 50% during profiling.
FIGI (#)COUNT_FIGISTRINGThe number of rows where the column matches the Financial Instrument Global Identifier format. Bigeye does not verify the check digit.
FIGI (%)PERCENT_FIGISTRINGThe number of rows where the column matches the Financial Instrument Global Identifier format. Bigeye does not verify the check digit. It is suggested as an autometric if Bigeye detects a match greater than 50% during profiling.
ISIN (#)COUNT_ISINSTRINGThe number of rows where the column matches the International Securities Identification Number format. Bigeye does not verify the check digit.
ISIN (%)PERCENT_ISINSTRINGThe percentage of rows where the column matches the International Securities Identification Number format. Bigeye does not verify the check digit. It is suggested as an autometric if Bigeye detect a match greater than 50% during profiling.

Time

Metric NameAPI NameColumn TypeDescription
Timestamp (#)COUNT_TIMESTAMP_STRINGSTRINGThe count of rows where the column matches an ISO-8601 date or timestamp format.
Timestamp (%)PERCENT_TIMESTAMP_STRINGSTRINGThe count of rows where the column matches an ISO-8601 date or timestamp format. It is suggested as an autometric if Bigeye detects a match >= 50% during profiling.
Not in Future (#)COUNT_NOT_IN_FUTURE
COUNT_DATE_NOT_IN_FUTURE
DATE_LIKE, TIMESTAMP_LIKEThe count of rows where the column contains a date or time that is not after the metric execution time.
Not in Future (%)PERCENT_NOT_IN_FUTURE
PERCENT_DATE_NOT_IN_FUTURE
DATE_LIKE, TIMESTAMP_LIKEThe percentage of rows where the column contains a date or time that is not after the metric execution time. It is suggested as an autometric if Bigeye detects a match >= 50% during profiling.

Geolocation

Column NameAPI NameColumn TypeDescription
Latitude (#)COUNT_LATITUDENUMERICThe count of rows where the column is a valid latitude; in the range between -90 and 90, inclusive.
Latitude (%)PERCENT_LATITUDENUMERICThe percentage of rows where the column is a valid latitude; in the range between -90 and 90, inclusive. It is suggested as an autometric if the column name contains lat and Bigeye detects a match greater than 80% during profiling.
Longitude (#)COUNT_LONGITUDENUMERICThe count of rows where the column is a valid latitude; in the range between -180 and 180, inclusive.
Longitude (%)PERCENT_LONGITUDENUMERICThe percentage of rows where the column is a valid latitude; in the range between -180 and 180, inclusive. It is suggested as an autometric if the column name contains long or lng and Bigeye detects a match greater than 80% during profiling.

User Specified

Metric NameAPI NameColumn TypeDescription
Value in list (%)PERCENT_VALUE_IN_LISTANYThe percentage of rows that match a user-supplied, comma-separated list of values. This metric is useful to validate fields with a small number of valid values