Optimizing Apache Spark clusters on EMR and Databricks to hit cost goals is a time-consuming task, but is an increasingly important task in today’s economy. Specifically for production workloads on AWS, optimizing instance types, spark configurations, and cluster sizes is an incredibly complex process. The Sync Autotuner for Apache Spark helps to solve this problem by predicting workload performance on alternative cloud infrastructure and configurations. In this talk developers will learn how to increase productivity, reduce cloud costs, and improve performance of the production Apache Spark workloads. In the second half we’ll walk users through a live hands-on demo of the Autotuner product.
Jeff received his PhD in EECS from UC Berkeley as an NDSEG fellow, was a Batelle Post-Doctoral Scholar at MIT, and is an Entrepreneurial Research Fellow at Activate. Prior to Sync, he was a technical staff member at MIT Lincoln Laboratory.
Kartik vaguely recalls writing a thesis on polyphase recursive all-pass filters for his masters in digital signal processing. Since then, he has switched gears and worked at several companies like Microsoft, Netflix and other startups specializing in Product Management.