To continually ensure accuracy in data replication and validate that the copied data from Table A to Table B match, leverage the Bigeye Deltas feature.
Deltas compare the contents of two tables and identify if any differences exist. The steps below outline how to create a Delta and effectively utilize it for monitoring:
Let us say we are replicating the
demo prod customer db into a
standard demo customer db table. To create a Delta over these tables, select the sources and the names for both these tables from the dropdowns.
Since we are replicating the tables, Deltas will map the columns between the tables automatically based on the column names. You can make adjustments if required.
You can also add filters to exclude certain rows from the Delta comparisons and add
Group Bys to understand column differences at a finer granularity.
You can give the Delta a representative name like Table A Replication Check.
By default, the Delta will run once and give you the comparison results. If your replication is a long process and you want to run the Delta checks periodically, you can also set a schedule to run it.
You can find more details about all of these steps in creating a Delta in Bigeye here.
After filling in all the details, click Create Delta.
Find the name of the Delta you just created in the list on the Deltas page. If you click on the name of a Delta you will see the details of the most recent run.
The header will give you a summary of details like the % match of row count between the tables you are trying to replicate and the number of column pairs where the metrics do not match.
The list shows each column pair, the metric, the metric value from each column, and the difference. If there is a difference, you will receive an alert. If the metric failed to run, it will show as Failed.
You can find more details about viewing Delta results here.
Updated about 2 months ago