Azure Data Factory (ADF) Lineage Connector

Overview

The Azure Data Factory (ADF) Lineage Connector enables teams to visualize and understand data movement and transformations within their ADF pipelines. It offers foundational support for key ADF components, with partial implementation reflecting high-priority use cases.

This document outlines what functionality is currently supported, and what broader ADF features are not yet implemented in the connector.

Supported Capabilities (Current State)

1. File-Based Lineage via ADF Project Files

• Users can export ADF metadata (pipelines, datasets, linked services, dataflows) using the download support files option from their Azure environment and run the connector on these files.
• No live ADF credentials required.
• Delivers the same results as direct REST API-based extraction.

2. REST API-Based Metadata Extraction (Optional)

For customers who do grant access, the connector can retrieve:

• Pipelines
• Linked services
• Datasets
• Dataflows

Metadata is modeled to support lineage generation.

3. Lineage for Incremental Copy Pipelines (Single Table)

Supports lineage across patterns like:

• Source DB ➝ Unconnected Lookup ➝ Activities ➝ Stored Procedure ➝ Target DB

Includes:

• Lookup activities (Table, Query, or Stored Procedure references)
• Visual lineage graphs built from supported components

Not Yet Implemented (Out of Scope in Current Release)

Complex Control Flows & Dynamic Execution

• ForEach activities with dynamic table lists
• Pipeline-to-pipeline calls

  These require runtime parameter resolution, not supported in the current release.

Data Transformations (Data Flows)

• Advanced transformations such as:
  • Join, Pivot, Unpivot, Aggregate, Conditional Split, Window, etc.

    These require additional testing and expression parsing.

File-Based Lineage (e.g., Amazon S3, flat files)

• Lineage for sources/sinks involving file-based storage (e.g., S3) is out of scope until FDS (File Data Store) support is implemented.

Intended Users & Usage

• Ideal for POC or limited-scope deployments
• Suitable for teams using basic pipeline activities or offline metadata extraction
• Not recommended for use cases requiring:
  • Full dynamic execution support
  • Parameterized pipeline resolution
  • Complex transformations

Future Capabilities (Planned)

Planned enhancements include:

• Lineage support for ForEach loops with dynamic tables
• Pipeline-to-pipeline lineage resolution
• Full ADF dataflow transformation parsing
• File-based lineage (e.g., S3, blob storage)

Questions?

If you’re unsure whether your ADF use case is supported, we’re happy to review your exported pipeline files and advise on compatibility.