Technical Talks

View All

Building a Control Plane for Data

Shirshanka Das Shirshanka Das | Co-Founder and CEO | Acryl Data

Keynote: March 28th @ 9:00am

Data Discovery, Data Observability, Data Quality, Data Governance, Data Management have traditionally been addressed as individual problems with distinct solutions. However, business challenges like access management, data retention, cost attribution and optimization, and end-to-end data quality require these tools to work together. Without a harmonizing layer across systems, these issues are impossible to manage in a uniform manner across the stack. We are calling this harmonizing layer the ‘Control Plane for Data’ - powered by the common thread across these systems: Metadata.  

In this talk, we’ll describe what the control plane of data looks like and how it fits into the reference architecture for the deconstructed data stack: a data stack that includes operational data stores, streaming systems, transformation engines, BI tools, warehouses, ML tools and orchestrators.

We’ll dig into the fundamental characteristics for a control plane:

  • Breadth (completeness)
  • Latency (freshness)
  • Scale
  • Source of Truth
  • Auditability
We’ll discuss what use-cases you can accomplish with a unified control plane and why this leads to a simpler, more flexible data stack. Finally, we’ll share thoughts on how the industry should evolve to enable this vision and how a constellation of projects including DataHub are working towards making this a reality. If you’re a data leader that has accumulated a ton of specialized data tools in a short amount of time and are wondering how to restore order back into your data teams’ daily lives, this talk is for you.

Shirshanka Das
Shirshanka Das
Co-Founder and CEO | Acryl Data

Shirshanka is co-founder and CEO of Acryl Data, the company which is commercializing the open source DataHub project, a real-time metadata platform used by LinkedIn, Stripe, Pinterest, Optum, Expedia and many others. Prior to founding Acryl, he was the overall architect for Big Data at LinkedIn from 2010 to 2020, and responsible for creating the metadata and data management strategy at the company. As part of this, he founded the DataHub project and shaped its evolution to a metadata platform that powers DataOps, MLOps, productivity, and governance use cases at LinkedIn. He is also a PMC and committer on the Apache Gobblin project which manages 100PB+ of data assets at rest at LinkedIn, and is deployed in production at other large companies like Verizon, PayPal etc. Prior to LinkedIn, Shirshanka worked on high-performance serving systems at Yahoo and PayPal. Shirshanka has a Ph.D. in Computer Science from UCLA.