Marquez: The complete OpenLineage solution

[amotl]

About

Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of dataset lifecycle management, and much more. Marquez was released and open sourced by WeWork.

Highlights

One Source of Truth

Marquez enables consuming, storing, and visualizing OpenLineage metadata from across an organization, serving use cases including data governance, data quality monitoring, and performance analytics.

One Service for Lineage

• Real-time metadata server
• Unified visual graph
• Flexible Lineage API

Resources

• https://github.com/MarquezProject/marquez
• https://marquezproject.ai/
• https://www.youtube.com/channel/UCeigRJGgeQ9Dn4yKWklG0rw
• https://openlineage.io/

[amotl]

Is it true that you may have worked with that application a while ago, @hlcianfagna? If you can recall the outcome, do you remember if it is suitable to be used together with CrateDB?

/cc @wierdvanderhaar

[hlcianfagna]

It is true, yes, and it was working fine with CrateDB, but we did not arrive to publish an article on it, let me share with you what I got by private message.

[amotl]

Wonderful. Thanks for your reply. Maybe @matkuliak could slot it in, to converge your outcome into a corresponding tutorial/documentation, in the same spirit like/after finishing crate/cratedb-guide#75?

[amotl]

Hi. I converted a document shared by @hlcianfagna (thanks!) into Markdown format, and added it as an article on the community forum, still in "unlisted" mode. It will need verification, possible updates, and a few formatting improvements. If someone wants to pick up from there...?

-- OpenLineage with Airflow, Marquez, and CrateDB

[amotl]

Hi @hlcianfagna. The article on the community forum referenced above is still in "unlisted mode". Shall we just publish it without much ado, consider this as "done", and add more iterations later, most likely by refactoring it into the CrateDB Guide?