Technical Talks

View All

Real-time Schema Discovery

Daniel Selans Daniel Selans | Co-founder & CTO | Streamdal

Nearly every data-engineer has had to deal with schema related issues. Product makes a change, backend adds some fields, data envelopes change and now your pipelines need to be updated. This is a painful reality that most data engineers deal with on a constant basis and is a significant time-waste in every data engineering org.

In this talk, I will show you how we developed a schema discovery process that is able to automatically evolve schemas in a complex distributed system that is processing upwards of a 100,000 messages per second.

I will dive deep into the details of schema versioning, detecting schema conflicts, compatibility and normalization, all without the use of any batching processes.
In this talk, I will show you how we developed a schema discovery process that is able to automatically evolve schemas in a complex distributed system that is processing upwards of a 100,000 messages per second.

I will dive deep into the details of how to detect schema drift, how to determine compatibility and ultimately how to do all of this, without having to involve batching.

Daniel Selans
Daniel Selans
Co-founder & CTO | Streamdal

Dan is the co-founder and CTO of Streamdal.com, a streaming data performance monitoring company. Dan previously wrote software at companies such as InVisionApp, New Relic and DigitalOcean and before that, spent over 10 years doing integration and R&D work at data centers.

Dan loves distributed systems and is convinced that the future of high performance is asynchronous.

He has been writing Go since 2014, works primarily in backend, listens to a lot of black metal and prefers Stella's over IPA’s. He resides in Portland, Oregon but is originally from [Riga, Latvia](https://goo.gl/maps/9wruLSg4RBU2).

FEATURED MEETINGS