Applying Design To Solve Scaling Problems and Evolve an Architecture

Beeswax is a high-scale internet-ad buying platform that processes over a million events per second, and differentiates from competitors by gathering and processing that event data and making it all available to customers. The business has grown quickly, forcing it to address scalability and flexibility issues with its existing data processing architecture.

Faced with resource and time constraints, Beeswax was able to both solve pressing operational concerns and create the architectural flexibility to add new data sources in the present and leverage new data processing technologies in the future by applying design patterns to the existing data processing technology stack. In this talk, we will describe the challenges, commonly occurring in data engineering, and approaches that we undertook to address them.