Explore integration with OpenLineage

[astrojuanlu]

OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used.

https://openlineage.io/

Reference implementation: Marquez https://github.com/MarquezProject/marquez

Validation:

With the release of Airflow 2.7.0, OpenLineage became a built-in feature of Airflow.

What does this mean for us? Should Kedro-Viz consume OpenLineage exports coming from Kedro? (See previous discussion at kedro-org/kedro-viz#1833 (comment))

[astrojuanlu]

A user asked

Is there any way to show log in Databricks for running pipelines/nodes execution such as time spending to complete specific node, memory usage, or cost in graphical manner like the image below?

I was just wondering there are any ways to visualize pipeline execution results/logs in graphical manner like for example, this node takes this much time to complete, etc.

Exporting OL data addresses this.

[astrojuanlu]

Another user:

What we are trying to capture and track is column-level spark transformations across the whole kedro pipeline

(hinted that only Kedro-related lineage would likely be insufficient)

[astrojuanlu]

OpenLineage allows solutions like DataHub to support more integrations. For example

DataHub's REST endpoint allows users to send OpenLineage events directly to DataHub.

https://datahubproject.io/docs/lineage/openlineage/

To automatically extract lineage information, the v2 plugin builds on top of Airflow's built-in OpenLineage extractors.

https://datahubproject.io/docs/lineage/airflow/#automatic-lineage-extraction

[astrojuanlu]

Well, here's a proof of concept!

https://github.com/astrojuanlu/kedro-openlineage

[astrojuanlu]

https://www.bearingnode.com/post/openlineage-vendor-integration-status

[astrojuanlu]

The "exploration" is done.

[astrojuanlu]

Now that we're working on visualising the run status and we plan to show live data kedro-org/kedro-viz#2310, I think there's some overlap with OpenLineage. @jitu5 @Huongg @rashidakanchwala should we do a spike on this after the MVP is released?