Migrate Metrics Across Data Sources

When you migrate metrics from a direct connection to an agent connection, you must have a valid migration path that helps you copy existing metrics to a new data source and backfill metrics. To move metrics between data sources, follow these steps:

  1. Authenticate with the Bigeye CLI.
    bigeye credential
    
  2. Export metrics from the existing data source to local JSON files.
    bigeye catalog get-metric-info --warehouse_id <source-warehouse-id> --output_path existing-directory/
    

    πŸ“˜

    To get the source warehouse ID, go to Catalog and click the data source from where you want to export the metrics. You can now copy the data source ID from the URL.

  3. Upsert metrics from the local JSON file to the target source warehouse.
    bigeye metric upsert-from-path -twid <target-warehouse-id> --source_path existing-directory/metric_info/warehouse_id-<source-warehouse-id> -srct -uc
    

    πŸ“˜

    • To get the target warehouse ID, go to Catalog and click the data source where you want to upsert the metrics. You can now copy the data source ID from the URL.
    • The -srct (Set Row Creation Time) parameter sets the row creation time on every table in the target warehouse to the row creation time used for those tables in the source warehouse.
    • The -uc (Update Collections) parameter adds the new metrics in the target warehouse to the existing collections used in the source warehouse. This parameter is optional, and when not used, you will have to rebuild Collections in the user interface.
  4. Backfill metrics.
    bigeye catalog backfill-metrics --warehouse_id <target-warehouse-id>
    
  5. Verify that all metrics are migrated from the source warehouse to the target warehouse. Once done, delete the source warehouse.
    bigeye catalog delete-source --warehouse_id <source-warehouse-id>
    

Migrating metrics must not be confused with editing metrics. When you migrate metrics from one data source to another, it simply means that you are copying them to a new warehouse. There can be some data loss during this process, such as:

  • Issues for metrics from the source warehouse.
  • Autothresholds feedback/training for metrics from the source warehouse.
  • Metric runs past the default backfill range (the default backfill is 21 days).
  • No migration of deltas .