Extinguishing the Garbage Fire of ML Testing

Emily Curtin, Staff MLOps Engineer | Intuit Mailchimp

Testing should be at the heart of Data Science. It is, after all, supposed to be Science: reproducible, reliable, consistent, all that. But testing for ML is a terrible experience. Nobody does it well, and some don't even do it at all. This leaves us all mired in a mess of unreliable models, rickety pipelines, and who-knows-what data in between.
 
As dutiful software engineers, most of us ML Eng or MLOps types have attempted to apply traditional software engineering methods to this domain, resulting in one of two extremes. Either we've achieved a degree of safety through overly-rigid test setups and CI requirements, or, whether through abdication or poor design, we've forced Data Scientists to throw caution to the wind and work around us to get anything done. 
 
Our traditional testing and CI methods are not working, but we can't just give up on providing guardrails. As engineers, how do we solve ML testing?
 
We solve it by doing what we do best: abstracting, decoupling, and separating concerns. We solve it by keeping pytest only where it belongs, which is not everywhere;by substituting testing for observability in appropriate places;by applying data reliability practices and thereby solving some problems at the source. 
 
Most importantly, we solve it by having deep customer empathy for our Data Scientists. By honoring our Data Scientists, their mental models, and their ways of working, we can work together to put out the garbage fire, but there may always be a smolder in the pile.

ABOUT THE SPEAKER

Emily Curtin

Staff MLOps Engineer | Intuit Mailchimp

Emily is a Staff MLOps Engineer at Intuit Mailchimp, meaning she gets paid to say "it depends" and "well actually." Professionally she leads a crazy good team focused on helping Data Scientists do higher quality work faster and more intuitively. Non-professionally she paints huge landscapes and hurricanes in oils, crushes sweet V1s (as long as they're not too crimpy), rides her bike, reads a lot, and bothers her cats. She lives in Atlanta, GA, which is inarguably the best city in the world, with her husband Ryan who's a pretty darn cool guy.

Emily Curtin