Creating an Extensible Big Data Platform

Creating an Extensible Big Data Platform
Reza Shiftehfar | Uber
DBT: Powerful, Open Source Data Transformations

DBT: Powerful, Open Source Data Transformations
Connor McArthur | Fishtown Analytics / DBT
Architecting the Data Lake: How to Ensure that Your ETL Pipelines Deliver High Quality Data

Architecting the Data Lake: How to Ensure that ...
Paul Lappas | Intermix
OASIS – Data Analysis Platform for Enterprise

OASIS – Data Analysis Platform for Enterprise
Keiji Yoshida | LINE Corporation
Building Highly Reliable Data Pipelines at Datadog

Building Highly Reliable Data Pipelines at Datadog
Quentin Francois | Datadog
Building a Feature Platform to Scale Machine Learning

Building a Feature Platform to Scale Machine ...
Willem Pienaar | GO-JEK
Creating a Data Engineering Culture

Creating a Data Engineering Culture
Jesse Anderson | Big Data Institute
Big Data Platform-as-a-Service for Cross-Media Monitoring

Big Data Platform-as-a-Service for Cross-Media ...
Liana Napalkova | Eurecat Technology Center
Apache Arrow: A Cross-language Development Platform for In-memory Data

Apache Arrow: A Cross-language Development ...
Wes McKinney | Ursa Labs, Member Apache Software Foundation
Computer Vision in Space

Computer Vision in Space
Marco Bressan | Satellogic
Easy Access to Data with Presto

Easy Access to Data with Presto
Iker Martinez de Apellaniz | Schibsted Classified Media
Event-Driven Data Architecture at Letgo

Event-Driven Data Architecture at Letgo
Ricardo Fanjul | Letgo
Automation of Machine Learning Workflows at BBVA

Automation of Machine Learning Workflows at BBVA
Jose Serrano | BBVA Data & Analytics
Flink SQL in Action

Flink SQL in Action
Timo Walther | dataArtisans, Committer & PMC Member of Apache Flink Project
Data Processing with Apache Beam: Towards Portability and Beyond

Data Processing with Apache Beam: Towards ...
Maximilian Michels | Google, PMC member of Apache Flink and Apache Beam
Using R in a Mid-Sized Data Analysis Scenario

Using R in a Mid-Sized Data Analysis Scenario
Richard Brosi | Yara Digital Labs
Driving in Dataland

Driving in Dataland
Carlos Herrera | Cabify
Marquez: A Metadata Service for Data Abstraction, Data Lineage, and Event-based Triggers

Marquez: A Metadata Service for Data Abstraction, ...
Willy Lulciuc | WeWork
Presto: Fast SQL-on-Anything

Presto: Fast SQL-on-Anything
Kamil Bajda-Pawlikowski | Starburst
Extract - Tiered Transform - Load (ETTL): A pipeline for a modular, scalable, and observable Internal Analytics platform

Extract - Tiered Transform - Load (ETTL): A ...
Jean-Mathieu Saponaro | Datadog
Using Embeddings to Understand the Evolution of Data Science Skill Sets

Using Embeddings to Understand the Evolution of ...
Maryam Jahanshahi | TapRecruit
Data Pipeline Frameworks: The Dream and the Reality

Data Pipeline Frameworks: The Dream and the ...
Mark Weiss | Beeswax
An Update on Scikit-learn

An Update on Scikit-learn
Andreas Mueller | Data Science Institute, Columbia University
Active Learning: Why Smart Labeling is the Future of Data Annotation

Active Learning: Why Smart Labeling is the Future ...
Jennifer Prendki | Alectio
Scaling Personalization via Machine-Learned Assortment Optimization

Scaling Personalization via Machine-Learned ...
Ethan Rosenthal | Dia&Co
Hindsight Bias: How to Deal with Label Leakage at Scale

Hindsight Bias: How to Deal with Label Leakage at ...
Till Bergmann | Salesforce
Oops I did it Again -- Adapting a Pop Music Identifier to Find Syndicated Content in Talk Radio

Oops I did it Again -- Adapting a Pop Music ...
Allison King | Cortico
Artwork Personalization at Netflix

Artwork Personalization at Netflix
Tony Jebara | Netflix
PyTorch 1.0 - The Platform for Accelerating AI Research to Production

PyTorch 1.0 - The Platform for Accelerating AI ...
Jeff Smith | Facebook AI Research
Scalability is Quantifiable: The Universal Scalability Law

Scalability is Quantifiable: The Universal ...
Baron Schwartz | VividCortex
Content Based Recommendations: Using Word Embeddings to Automate Related Content Generation at BuzzFeed

Content Based Recommendations: Using Word ...
Carolyn Huangci | BuzzFeed
Building a Modern Machine Learning Platform on Kubernetes

Building a Modern Machine Learning Platform on ...
Saurabh Bajaj | Lyft
Evolving Stitch Fix's Data Platform for Data Lineage

Evolving Stitch Fix's Data Platform for Data ...
Neelesh Srinivas Salian | Stitch Fix
The Literate Programmer: Cargo Cult Open Source

The Literate Programmer: Cargo Cult Open Source
Wes Chow | Cortico
Accelerating Single-cell Bioinformatics with N-dimensional Arrays in the Cloud

Accelerating Single-cell Bioinformatics with ...
Ryan Williams | Icahn School of Medicine at Mount Sinai
The Software Architecture of WayUp's Job Recommender System

The Software Architecture of WayUp's Job ...
Harlan Harris | WayUp
Automating Modeling Pipelines

Automating Modeling Pipelines
William Nelson | Intent Media Inc
Building Data Tools that Work

Building Data Tools that Work
Benn Stancil | Mode
Analyzing Data in the Cloud: Is True Privacy and Security Possible?

Analyzing Data in the Cloud: Is True Privacy and ...
Raghu Murthy | Datacoral
Stream Processing Design Patterns

Stream Processing Design Patterns
Andreas Markmann | Capital One
Scale Processes, Not People: How Data Teams Do More With Less By Adopting Software Engineering Best Practices

Scale Processes, Not People: How Data Teams Do ...
Thomas La Piana | GitLab
AI farming: 100x the yield with a data team of 1

AI farming: 100x the yield with a data team of 1
Sam Swift | Bowery Farming
The Highs and Lows of Building an Adtech Data Pipeline

The Highs and Lows of Building an Adtech Data ...
Dan Goldin | TripleLift
The Customer as The Unit of Analysis: Models, Metrics and a Multitude of Uses

The Customer as The Unit of Analysis: Models, ...
Brian Bloniarz | Second Measure
Technical Founders Panel

Technical Founders Panel
William Falcon | NextGenVest
Computer Vision AI to Disrupt Digital Advertising

Computer Vision AI to Disrupt Digital Advertising
Joy Tang | Markable AI
Running Effective Machine Learning Teams: Common Issues, Challenges and Solutions

Running Effective Machine Learning Teams: Common ...
Gideon Mendels | Comet.ml
Optimizing Time to Data through Streams and Data Abstraction

Optimizing Time to Data through Streams and Data ...
Nicolas Joseph | Datalogue
Engineering Lessons Learned by Data Scientists in Growing MalwareScore from Kaggle Competition to Trusted Antivirus Solution

Engineering Lessons Learned by Data Scientists in ...
Phil Roth | Endgame
The Unreasonable Deceptiveness of Bad Data

The Unreasonable Deceptiveness of Bad Data
Rigel Swavely | Clarifai
Fixing the Big Data Development Cycle with SQL

Fixing the Big Data Development Cycle with SQL
Justin Coffey | Criteo Labs
Building a Knowledge Graph Platform from Billions of Unstructured Online Sources Using AI

Building a Knowledge Graph Platform from Billions ...
Aditya Jami | Meltwater
Three Tips for Better Predictive Modeling

Three Tips for Better Predictive Modeling
Stephanie Yang | Foursquare
AI Challenges in Customer Care Automation

AI Challenges in Customer Care Automation
Sameer Yami | Linc Global
Building a Music Analytics Pipeline at Pandora

Building a Music Analytics Pipeline at Pandora
Brian Femiano | Pandora
Fast Data apps with Alpakka Kafka connector and Akka Streams

Fast Data apps with Alpakka Kafka connector and ...
Sean Glover | Lightbend, Inc
Causal Data Science

Causal Data Science
Adam Kelleher | Barclays Investment Bank
Machine Learning from Development to Production at Instacart

Machine Learning from Development to Production ...
Montana Low | Instacart
Cloud Data Warehouse Benchmark: Redshift vs Snowflake vs BigQuery

Cloud Data Warehouse Benchmark: Redshift vs ...
George Fraser | Fivetran
Democratizing Data with the Clover Transform Framework

Democratizing Data with the Clover Transform ...
Chris Hartfield | Clover Health
Functional Data Engineering - A Set of Best Practices

Functional Data Engineering - A Set of Best ...
Max Beauchemin | Stealth
Uber’s Data Journey: 100+PB with Minute Latency

Uber’s Data Journey: 100+PB with Minute Latency
Reza Shiftehfar | Uber
Scaling a Relational Database for the Cloud-age

Scaling a Relational Database for the Cloud-age
Sumedh Pathak | Citus Data
Lazy Beats Smart and Fast

Lazy Beats Smart and Fast
Julian Hyde | Looker
Effective Management of High Volume Numeric Data with Histograms

Effective Management of High Volume Numeric Data ...
Fred Moyer | Circonus
From Flat Files to Deconstructed Database: The Evolution and Future of the Big Data Ecosystem

From Flat Files to Deconstructed Database: The ...
Julien Le Dem | WeWork
A Trillion Rows Per Second as a Foundation for Interactive Analytics

A Trillion Rows Per Second as a Foundation for ...
Eric Hanson | MemSQL
What the heck is an In-Memory Data Grid?

What the heck is an In-Memory Data Grid?
Addison Huddy | Pivotal
Data Access for Data Science

Data Access for Data Science
Jacques Nadeau | Dremio
A Multi-Armed Bandit Framework for Recommendations at Netflix

A Multi-Armed Bandit Framework for ...
Jaya Kawale | Netflix
Fast & Effective: Natural Language Understanding

Fast & Effective: Natural Language Understanding
Mike Conover | Workday
Weld: Accelerating Data Science by 100x

Weld: Accelerating Data Science by 100x
Shoumik Palkar | Stanford University Infolab
AutoML: The Assembly Line of Machine Learning

AutoML: The Assembly Line of Machine Learning
Mayukh Bhaowal | Salesforce
Enabling Full Stack Data Scientists at Stitch Fix

Enabling Full Stack Data Scientists at Stitch Fix
Juliet Hougland | Stitch Fix
Safely Streamlining Healthcare Policy Management using Ideas from Structured Natural Language Processing (SNLP)

Safely Streamlining Healthcare Policy Management ...
Asif Khalak | Collective Health
Democratizing Metric Definition and Discovery at Airbnb

Democratizing Metric Definition and Discovery at ...
Lauren Chircus | Airbnb
Define Once, Evaluate Anywhere: Building Repeatable and Correct Features at Stripe

Define Once, Evaluate Anywhere: Building ...
Kelley Rivoire | Stripe
Hazardous Models and Risk Mitigation in Real Estate

Hazardous Models and Risk Mitigation in Real ...
David Lundgren | Opendoor
The Design of Systems for Real-time Prediction Serving

The Design of Systems for Real-time Prediction ...
Joseph Gonzalez | UC Berkeley
Data Science: Past, Present, Future

Data Science: Past, Present, Future
Shubha Nabar | Salesforce
VC Panel Talk

VC Panel Talk
Lisha Li | Amplify PartnersAmplify Partners
Macrobase: A Search Engine for Fast Data Streams

Macrobase: A Search Engine for Fast Data Streams
Sahaana Suri | Stanford University, Future Data Systems
TimescaleDB: Rearchitecting a SQL database for time-series data

TimescaleDB: Rearchitecting a SQL database for ...
Mike Freedman | TimescaleDB
Production Analytics With a Distributed Column Store

Production Analytics With a Distributed Column ...
Sam Stokes | Honeycomb
Don’t optimize my queries, optimize my data!

Don’t optimize my queries, optimize my data!
Julian Hyde | Looker
Worse Case Scenario in the Database

Worse Case Scenario in the Database
Marianne Bellotti | United States Digital Service
The Challenges of Distributing Postgres: A Citus Story

The Challenges of Distributing Postgres: A Citus ...
Ozgun Erdogan | Citus Data
The Statistics of Dirty Data

The Statistics of Dirty Data
Sanjay Krishnan | UC Berkeley, AMPLab
Easy, Scalable, Fault-tolerant Stream Processing with Structured Streaming in Apache Spark

Easy, Scalable, Fault-tolerant Stream Processing ...
Burak Yavuz | Databricks
Using Apache Arrow, Calcite and Parquet to build a Relational Cache

Using Apache Arrow, Calcite and Parquet to build ...
Jacques Nadeau | Dremio
How Spotify Distills Terabytes of Raw Data into Meaningful Music Recommendations

How Spotify Distills Terabytes of Raw Data into ...
Gandalf Hernandez | Spotify
Building a Recursive BigQuery Mapper

Building a Recursive BigQuery Mapper
Darren McCleary | The New York Times
Building ETL Infrastructure that Analysts Love

Building ETL Infrastructure that Analysts Love
Christian Romming | ETLeap
Using Apache Spark for processing trillions of records each day at Datadog

Using Apache Spark for processing trillions of ...
Vadim Semenov | Datadog
Productizing structural models

Productizing structural models
James Savage | Lendable
Lessons in hiring data scientists

Lessons in hiring data scientists
Genevieve Smith | Insight
Using Causal Forests for Subpopulation Identification in Randomized Clinical Trials

Using Causal Forests for Subpopulation ...
James Faghmous | Icahn School of Medicine at Mt. Sinai
You Won't Believe How We Optimize our Headlines

You Won't Believe How We Optimize our Headlines
Lucy Wang | BuzzFeed
Deploying Data Science for Distribution of The New York Times

Deploying Data Science for Distribution of The ...
Anne Bauer | The New York Times
Zip codes and other lies your map told you

Zip codes and other lies your map told you
Peter Lenz | Dstilery
Automating machine learning

Automating machine learning
Andreas Mueller | Data Science Institute, Columbia University
Building automated support at Kickstarter

Building automated support at Kickstarter
Jeffrey Doker | Kickstarter
Privacy Techniques for Data Science

Privacy Techniques for Data Science
Jim Klucar | Immuta
The Future of Data Science in the Media

The Future of Data Science in the Media
Haile Owusu | Mashable
Composable Interfaces for Parallel Processing in Apache Spark & Weld

Composable Interfaces for Parallel Processing in ...
Matei Zaharia | Databricks
Data Science @ Pinterest

Data Science @ Pinterest
Mohammad Shahangian | Pinterest
Data Science in the Enterprise

Data Science in the Enterprise
Sean Anderson | Cloudera
Practical Solutions for Annoying Machine Learning Problems

Practical Solutions for Annoying Machine Learning ...
Alyssa Frazee | Stripe
Beyond 50,000 Partitions: How Heroku Pushes the Limits of Kafka at Scale

Beyond 50,000 Partitions: How Heroku Pushes the ...
Jeff Chao | Heroku
Scaling Up Spark at Facebook – a 60TB Production Use Case

Scaling Up Spark at Facebook – a 60TB Production ...
Shuojie Wang | Facebook
How Superset and Druid Power Real-Time Analytics at Airbnb

How Superset and Druid Power Real-Time Analytics ...
Max Beauchemin | Stealth
Hoodie: An Open Source Incremental Processing Framework From Uber

Hoodie: An Open Source Incremental Processing ...
Vinoth Chandar | Uber
Data for the 99%

Data for the 99%
Benn Stancil | Mode
Payment Fraud in Digital Currency

Payment Fraud in Digital Currency
Soups Ranjan | Revolut
The Limitations of Big Data in Predictive Analytics

The Limitations of Big Data in Predictive ...
Jennifer Prendki | Alectio
Practical Lessons for Building Machine Learning Models in Production

Practical Lessons for Building Machine Learning ...
Sharath Rao | Instacart
An Introduction to Big Data's Unsung Hero: The Log

An Introduction to Big Data's Unsung Hero: The Log
Liz Bennett | Stitch Fix
Why, When, How: Lessons Learned in Applying Deep Learning to Real-World Problems

Why, When, How: Lessons Learned in Applying Deep ...
Daniel Galron | eBay
Anomaly Detection for Data Quality and Metric Shifts at Netflix

Anomaly Detection for Data Quality and Metric ...
Laura Pruitt | Netflix
Cloud-Native Stream Processing

Cloud-Native Stream Processing
Sid Anand | Agari
Parsing of Diverse Healthcare Data

Parsing of Diverse Healthcare Data
Chris Hartfield | Clover Health
Interactive Exploratory Analytics with Druid

Interactive Exploratory Analytics with Druid
Fangjin Yang | Imply
InfluxDB Storage Engine Internals

InfluxDB Storage Engine Internals
Paul Dix | InfluxData
A Nation of Immigrants: The Data Sciences

A Nation of Immigrants: The Data Sciences
Kenneth Sanford | Dataiku
Building 21st Century Data Science and Data Engineering Teams

Building 21st Century Data Science and Data ...
Polong Lin | IBM
Twitter Heron: The Path Towards Elastic Streaming

Twitter Heron: The Path Towards Elastic Streaming
Ashvin Agrawal | Microsoft
Simulation-based Inference: Advantages Over A/B Testing in Real Estate

Simulation-based Inference: Advantages Over A/B ...
Nelson Ray | Opendoor
Format Wars: from VHS and Beta to Avro and Parquet

Format Wars: from VHS and Beta to Avro and Parquet
Silvia Oliveros Torres | Silicon Valley Data Science
Real-time System Computing Engines

Real-time System Computing Engines
Steffen Peter | Levyx
The Right Stuff: Lessons Learned from a Decade of Data Engineering

The Right Stuff: Lessons Learned from a Decade of ...
Ben Hamner | Kaggle
How Engineer Angels Evaluate Data-Oriented Companies

How Engineer Angels Evaluate Data-Oriented ...
Jocelyn Goldfein | Zetta
The Future of Column-Oriented Data Processing with Arrow and Parquet

The Future of Column-Oriented Data Processing ...
Julien Le Dem | WeWork
Causal Inference in Data Science From Prediction to Causation

Causal Inference in Data Science From Prediction ...
Amit Sharma | Microsoft Research
Reinforcement Learning for Data Scientists

Reinforcement Learning for Data Scientists
Brian Farris | Bloomberg
How to Change a City with Data Science

How to Change a City with Data Science
Ben Wellington | Two Sigma
Python Data Wrangling: Preparing for the Future

Python Data Wrangling: Preparing for the Future
Wes McKinney | Ursa Labs, Member Apache Software Foundation
Lessons Learned Optimizing NoSQL for Apache Spark

Lessons Learned Optimizing NoSQL for Apache Spark
John Musser | Ford Motor Company
Unified Pipeline Architecture: The Evolution of Data Processing at Spotify

Unified Pipeline Architecture: The Evolution of ...
Erin Palmer | Spotify
SystemML & Spark: a Framework for Scalable Data Science Algorithm Development

SystemML & Spark: a Framework for Scalable Data ...
Jerome Nilmeier | IBM
Stop Obsessing about Data Infrastructure

Stop Obsessing about Data Infrastructure
Yair Weinberger | Alooma
Apache Kafka and the Rise of Stream Processing

Apache Kafka and the Rise of Stream Processing
Guozhang Wang | Confluent
Genomic Data Analysis with Spark & Hadoop

Genomic Data Analysis with Spark & Hadoop
Ryan Williams | Icahn School of Medicine at Mount Sinai
Anomaly Detection for Real-World Systems

Anomaly Detection for Real-World Systems
Manojit Nandi | STEALTHbits
Building a Cloud-Native SQL Database

Building a Cloud-Native SQL Database
Alex Robinson | Cockroach Labs
To Get the Value, Ditch the Hype

To Get the Value, Ditch the Hype
Nick Ursa | The New York Times
Statistical and Computational Challenges of Real-Time News Clustering

Statistical and Computational Challenges of ...
Jeiran Jahani | Chartbeat
Predicting Chaotic Systems with Sparse Data: Lessons from Numerical Weather Prediction

Predicting Chaotic Systems with Sparse Data: ...
David Kelly | New York University
The Trials and Tribulations of Scaling Data Science and Engineering

The Trials and Tribulations of Scaling Data ...
Ashley Miller | Datadog
Peloton: The Self-Driving Database Management System

Peloton: The Self-Driving Database Management ...
Andy Pavlo | Carnegie Melon University
Elastic Big Data Platform at Datadog

Elastic Big Data Platform at Datadog
Doug Daniels | Datadog
Processing Geographic Data at Internet Scale

Processing Geographic Data at Internet Scale
Peter Lenz | Dstilery
Bias, Variance and Adaptive Products

Bias, Variance and Adaptive Products
George Davis | Frame.ai
Career Panel - Leveling Up in Your Career as a Data Scientist/Engineer

Career Panel - Leveling Up in Your Career as a ...
Nick Chamandy | Lyft
VC Panel - The Present Future of Data-Oriented Startups | DataEngConf NY '16

VC Panel - The Present Future of Data-Oriented ...
Matt Hartman | Betaworks
Actionable and Interpretable Predictions from a Stacked Model

Actionable and Interpretable Predictions from a ...
Austen Head | Quid
Scalability! But at What COST?

Scalability! But at What COST?
Frank McSherry |
Reducing Student Loans with Bot-Powered Humans

Reducing Student Loans with Bot-Powered Humans
William Falcon | NextGenVest
Building Bots, Building Blocks: How Forbes Experiments, Evaluates, and Kills Data-driven Bots

Building Bots, Building Blocks: How Forbes ...
Luis Capelo | Forbes
Real-Time Data Pipelines Made Easy with Structured Streaming in Apache Spark

Real-Time Data Pipelines Made Easy with ...
Tathagata Das | Databricks