Technical Talks

View All

Evolving Data Pipelines at Scale

Iaroslav Zeigerman Iaroslav Zeigerman | Co-founder & Chief Architect | Tobiko

When it comes to developing new data pipelines and evolving existing ones, the available tooling lags behind what's available to software developers. Creating new development environments is cumbersome, populating these environments with representative data is compute-intensive, and the process of deploying changes to production is ad-hoc and error-prone. All of these factors drive up development costs, slow down iteration speed, and make data less reliable.

At the core of the open-source project SQLMesh is a novel development workflow, born from our collective experience at companies like Airbnb, Apple, Google, and Netflix. It is designed to handle the complexities of evolving data pipelines at an internet scale. In this talk, we’ll discuss challenges faced by data practitioners today and highlight core SQLMesh concepts addressing them, including virtual data environments, automatic change categorization, continuous testing, and accurate change previews.

Iaroslav Zeigerman
Iaroslav Zeigerman
Co-founder & Chief Architect | Tobiko

Iaroslav Zeigerman has over a decade of experience in building data, ML, and experimentation platforms. He currently leads the development of SQLMesh, the open-source data transformation platform at Tobiko Data. Before co-founding Tobiko Data, he led data platform and engineering efforts at companies like Apple and Netflix, designing solutions that address challenges of building and evolving data pipelines at an internet scale. Iaroslav is one of the core contributors to SQLGlot and the creator of m2cgen.