The biggest initial hurdle to success with Big Data isn’t technical - it’s management. Your data engineering project’s initial success is predicated on your management team correctly staffing and resourcing it. This runs opposite to how most data engineering teams are started and run. If you just choose the best technologies, things will just fall into place. They don’t and that’s a common pattern for failure.
But how do you correctly do something that’s so new? This could be your team’s first data engineering project. What should the team look like? What skills should the team have? What should you look for in Data Engineer (because you’ll probably have to hire a Software Engineer and train them)? What are some of the management pitfalls?
In this talk, we will cover the most common reasons why data engineering teams fail and how to correct them. This will include ways to get your management to understand that data engineering is really complex and time consuming. It is not data warehousing with new names. Management needs to understand that you can’t compare a data engineering team to the web development team, for example.
Jesse will share the stories of teams who haven’t set up their data engineering culture correctly and what happened. Then, Jesse will talk about the teams who’ve turned around their culture and how they did it. FInally, Jesse will share the skills that every data engineering team needs.
Jesse Anderson is a Data Engineer, Creative Engineer and Managing Director of Big Data Institute. He trains at companies ranging from startups to Fortune 100 companies on Big Data. This includes training on cutting edge technology like Apache Kafka, Apache Hadoop and Apache Spark. He has taught thousands of students the skills to become Data Engineers. He is widely regarded as an expert in the field and his novel teaching practices. Jesse is published on O’Reilly and Pragmatic Programmers. He has been covered in prestigious publications such as The Wall Street Journal, CNN, BBC, NPR, Engadget, and Wired.