We have seen great advancements and innovations across the data management landscape - from large-scale processing engines to cloud-based data warehouses and even out to BI and machine learning tools. But why is it that the most powerful systems of today are being controlled and orchestrated by some of the most rudimentary technologies? How data moves and is transformed is still dictated by manual, hard-coded triggers and rules, resulting in slow development cycles and brittle pipelines. For many data engineering teams, this also causes a high maintenance burden and disproportionate amount of time spent combing through code and logs, and constantly tuning parameters just to keep things running. As the amount of pipelines and dependencies scale, these challenges compound and eclipse what can be manually constructed and managed by even the most talented data engineering teams.
This is what led us to rethink the state of pipeline orchestration. In taking a page from other technologies, such as Kubernetes, React, and even compilers, we opted to move away from imperative, task-based design and move to declarative, data-centric automation. In this session, Sean Knapp, CEO & founder of Ascend, will discuss what led us to take this approach and essentially rearchitect pipeline orchestration from the ground up. In particular, he will talk through:
- Current assessment of orchestration tools and advancements
- Tradeoffs and design decisions in moving from task-centric to data-centric architectures
- Challenges Ascend faced running a massive-scale, elastic infrastructure built on Kubernetes and Apache Spark
- How Ascend built a Control Plane to combine declarative configurations and automation to power intelligent orchestration.
Sean Knapp is the founder and CEO of Ascend.io.
Prior to Ascend.io, Sean was a co-founder, CTO, and Chief Product Officer at Ooyala. At Ooyala Sean played key roles in raising $120M, scaling the company to 500 employees, Ooyala's $400m+ acquisition, as well as Ooyala's subsequent acquisitions of Videoplaza and Nativ. He oversaw all Product, Engineering and Solutions, as well as defined Ooyala's product vision for their award winning analytics and video platform solutions.
Before founding Ooyala, Sean worked at Google where he was the technical lead for Google's legendary Web Search Interface team, helping that team increase Google revenues by over $1B. Sean also developed and launched iGoogle, the company's popular, customizable home page. Sean has both B.S. and M.S. degrees in Computer Science from Stanford University.