Over the last decade I have been lucky enough to contribute a few successful open source projects to the data ecosystem. In this talk I’ll share the story of how these projects came to be and what made their success possible. I’ll describe the ideation process and early growth of the Apache Parquet columnar format and show how this led to the creation of its in-memory alter-ego Apache Arrow. I’ll end with showing how this experience enabled the success of OpenLineage, an LFAI & Data project that brings observability to the data ecosystem. Along the way, I’ll talk about the key elements that catalyzed their growth, from project focus to governance to community.
Julien Le Dem is the OpenLineage project lead at the LFAI&Data. He co-created the Parquet file format and is involved in several open source projects including OpenLineage, Marquez, Arrow, Iceberg and a few others. He was the Chief Architect of Astronomer and Co-Founded Datakin. Previously he held technical leadership positions at Wework, Dremio on the founding team, Twitter, where he also obtained a two-character Twitter handle (@J_); and Yahoo!, where he received his Hadoop initiation. His French accent makes his talks particularly attractive.