Measuring & aggregating network request duration percentiles and error rate are essential for Uber teams to monitor the reliability of network libraries and catch performance issues, at a global scale across 600+ cities/4500+ mobile carriers. In this talk, we will discuss how we used the Hudi Delta Streamer to build Spark pipelines to incrementally pull data sources and generate network metrics, powering near real-time network dashboards.
Hudi is an open-source data format managing storage of large analytical datasets, providing efficient upsert and incremental processing primitives. We will describe the design of incrementally updating different types of network metrics using Hudi upserts and how it reduces ingestion and processing latency by taking advantage of incremental pulls. We will also share our experience on launching and running near real-time network analytics pipelines at Uber.
Nishith Agarwal is a senior software engineer at Uber, where he works on the Hudi project and the Hadoop platform at large. His interests lie in large-scale distributed and data systems.
Ethan is a Software Engineer at Uber. Previously he worked at Google as Software Engineering Intern. Ethan finished his Ph.D & Masters from University of Michigan.