Technical Talks

View All

Unified Pipeline Architecture: The Evolution of Data Processing at Spotify

Erin Palmer Erin Palmer | Applied Data Scientist | Spotify

In Spotify Creator we strive to provide artists an accurate view of what their fan base is. We crunch numbers for 100 Million platform monthly active users daily and compute dashboards for millions of artists. As a result we process terabytes of data daily and condense it to about ~200 GB of data. In this talk we will discuss the evolution of our pipelines and how we made them more resilient to the irregularities of data as well as external failures.

Erin Palmer
Erin Palmer
Applied Data Scientist | Spotify

Erin Palmer is an applied data scientist at Spotify and software engineer with over 6 years industry experience and a Masters degree in Machine Learning and Computer Vision. Previously, she worked at Google on the AdWords platform making quality improvements in the automatic bidding space, as well as at FactSet, where she worked primarily on the backend of the Portfolio Analytics application.