Your organization is growing. New internal clients, consumers, interfaces, integrations, APIs are joining by the day. Each one of them is controlled by a different service, a different team, a different format, and the hardest one - a different model. The path towards upstream breaks, consumer crashes, unreliable data, and desprate data engineers is set. Enforcing a standard on that point-in-time is complicated and almost imposible as there are probably too deep production dependencies, thats why it is crucial to lay the foundations to data governance from day 1. In this talk, you will learn what are the risks of growing without data governance, how to create a supportive framework to control your different data clients using a common open-source tools, and how to consruct the framework to adapt to changes and additions that scale with the business movement.
My name is Yaniv Ben Hemo. Living in Israel, married to Nicole, father of Ari and Balu - our Golden Retriever. I started to code when I was ten years old, and ever since, I have been doing my best to learn, evolve, and build. In recent years, I have been fascinated by data and its power to disrupt our world, which made me deepen my journey with it, specifically, data engineering. On January 5th, 2022, alongside my three best friends from college, we founded Memphis.dev, a real-time data processing platform out of our struggles as devs working with legacy queues and brokers to enable high-scale real-time data processing for developers.