Using GIT as a NoSQL Database for Fine-Grained Control Over the Data Pipeline

In this talk, Hung will reveal how Y42, an all-in-one data pipeline tool, has leveraged Git as a noSQL database to foster unparalleled collaboration opportunities between data engineers and data analysts.

Y42’s decision to abandon all classical databases within their platform in favor of using Git has brought a series of major innovations — but this cutting-edge approach hasn’t been without challenges. This talk will highlight the extraordinary benefits of using Git-as-a-database to power an all-in-one data workspace, such as: 

• Data-pipeline-as-code (including dashboard-as-code and integrations-as-code)

• Browser-based, high-performance implementation of Git using WebAssembly, meaning the Y42-data-pipeline can be implemented using no-code, low-code and/or code

• Simple end-to-end templating

• Easy version control + rollback + environment of job status, settings and data warehouse tables

• Coherent pipeline automations with one orchestration layer

• And it doesn’t end there…

However, deciding to use this unconventional approach meant they were in for a bumpy ride. The Y42 team found themselves facing a series of challenges, including: 

• Performance issues to save thousands of jobs inside Git

• Access control issues using folder paths as ids

• Getting the web app to seamlessly integrate Git and code 

The Y42 team invites you to attend this talk and learn more about using Git as a noSQL database for a new data revolution. It’s a bold statement, but they believe this talk has the potential to significantly shape the data industry for the years to come.