Technical Talks

View All

Beyond Kafka: Cutting Costs and Complexity with WarpStream and S3

Ryan Worl Ryan Worl | Co-founder & CTO | WarpStream

Apache Kafka powers many event and ETL pipelines, but it is notoriously difficult and expensive to operate in the cloud. Whether you’re dealing with a high-throughput workload that costs an arm and a leg in cross-AZ data transfer, a long retention duration that explodes your EBS bill, or a cluster that pages you every time a node fails because you need to rebalance partitions: Kafka leaves you without many options.

This workshop will explain how building streaming storage on top of object storage like Amazon S3 mitigates all three of these pain points, and how we built WarpStream to package these ideas into an Apache Kafka protocol-compatible interface. We’ll benchmark a WarpStream cluster running live with a workload of hundreds of MiB/s of ingestion and compare the TCO to running open-source Apache Kafka yourself.

Ryan Worl
Ryan Worl
Co-founder & CTO | WarpStream

Ryan is a distributed systems and databases nerd. His favorite weekend pastime is staying up to date on the latest database internals and consensus protocols research publications. At Datadog, Ryan conceptualized the idea for Husky and designed the original architecture. Subsequently he worked with Richie to make Husky a reality and migrated many of Datadog's products to it.