SQL adoption is growing quickly thanks to modern SQL warehouses and SQL technologies, enabling teams to do more and more with their data without having to write and maintain imperative code. Despite this, SQL still has many limitations which make it hard to adopt software engineering best practices such as unit testing, assertions, releases, and writing modular, maintainable code bases.
In this talk we take a look at what metaprogramming is and how it can help us build on top of SQL to address these shortcomings, as well as unlock a number of new and interesting use cases. We will walk through the metalanguage and framework we have developed at Dataform, the challenges we faced, the design decisions we made, and many real world examples of things you probably never thought you could (or should) do with SQL.
Lewis Hemens is the CTO & co-founder of Dataform (YC W18). Previously he worked at Google as an Engineering Manager and before that as a Software Engineer, where he developed large scale data products within the Google Display Network, including the AdSense recommendation and A/B experimentation platforms. He holds a BA in Computer Science from Cambridge University.