Batch data processing (also known as ETL) is time-consuming, brittle, and often unrewarding. Not only that, it’s hard to operate, evolve, and troubleshoot.
In this talk, we’ll discuss functional programming paradigm and explore how applying it to Data Engineering can bring a lot of clarity to the process. It helps solving some of the inherent problems of ETL, leads to more manageable and maintainable workloads and helps to implement reproducible and scalable practices. It empowers data teams to tackle larger problems and push the boundaries of what’s possible.
Maxime Beauchemin recently joined Lyft as a Software Engineer after some time at Airbnb as a data engineer developing tools to help streamline and automate data-engineering processes. He is also the creator and lead committer on Apache Airflow and Apache Superset. He mastered his data-warehousing fundamentals at Ubisoft and was an early adopter of Hadoop/Pig while at Yahoo in 2007. More recently, at Facebook, he developed analytics-as-a-service frameworks around engagement and growth-metrics computation, anomaly detection, and cohort analysis. He’s a father of three, and in his free time, he’s a digital artist.