Beeswax is a high-scale internet-ad buying platform that processes over a million events per second, and differentiates from competitors by gathering and processing that event data and making it all available to customers. The business has grown quickly, forcing it to address scalability and flexibility issues with its existing data processing architecture.
Faced with resource and time constraints, Beeswax was able to both solve pressing operational concerns and create the architectural flexibility to add new data sources in the present and leverage new data processing technologies in the future by applying design patterns to the existing data processing technology stack. In this talk, we will describe the challenges, commonly occurring in data engineering, and approaches that we undertook to address them.
Mark Weiss is a Senior Software Engineer at Beeswax, the online advertising industry’s first extensible programmatic buying platform, where he focuses on designing and building data processing infrastructure and applications supporting reporting and machine learning. He has previously held various engineering individual contributor and leadership roles, and has worked on ETL systems and data-driven distributed platforms for much of his career. Mark has spoken previously at DataEngConf NYC, and regularly speaks and mentors at the NYC Python Meetup. He is also blogs and hosts the podcast "Using Reflection" at http://www.usingreflection.com, and can be found on Github, Twitter and LinkedIn under @marksweiss. He lives in Brooklyn, NY