Pitfalls and Challenges of ML-Powered Applications

As a field, we often hear about success stories. This is true in research, where a publishing incentive can pressure authors to focus on consistently exceeding state of the art results. It is also true in industry, where companies attempt to attract engineering talent by describing how impressive their production ML systems are.

However, every practitioner here knows that in engineering and in ML, the road to success is paved with failures. The field of ML in production is new, and so has a lack of cautionary tales of things that can go wrong with models. This talk will try to help correct that.

We will discuss challenges such as performance mismatch between offline training and online inference, feature generation and data leakage, and adequate roadmap planning for ML.