At the beginning of 2021, Shopify Data Platform decided to adopt Apache Flink to enable modern stateful stream-processing. Shopify had a lot of experience with other streaming technologies, but Flink was a great fit due to its state management primitives.
After about six months, Shopify now has a flourishing ecosystem of tools, tens of prototypes from many teams across the company and a few large use-cases in production.
Yaroslav will share a story about not just building a single data pipeline but building a sustainable ecosystem. You can learn about how they planned their platform roadmap, the tools and libraries Shopify built, the decision to fork Flink, and how Shopify partnered with other teams and drove the adoption of streaming at the company.
Yaroslav Tkachenko is a software engineer interested in distributed systems, microservices, data-intensive applications, functional programming, modern cloud infrastructure, and DevOps practices. Currently, Yaroslav is a Staff Data Engineer at Shopify, working on data pipelines for merchant-facing data products and analytics. Previously, he was a Senior Software Engineer and later Software Architect at Activision, where he redesigned and rebuilt the data pipeline for Activision games like Call of Duty franchise. Before joining Activision, Yaroslav held various leadership roles in multiple startups. He was responsible for designing, developing, delivering and maintaining platform services and cloud infrastructure for mission-critical systems.