Astronomer is a data engineering platform that collects, processes and unifies an organization’s data so that they can get straight to analytics and data science. Initially, Astronomer built their pipeline on top of AWS-managed services to get moving quickly, but ran into some limitations down the road.
In this talk, Astronomer outlines their transition to open source software like Mesos, Docker, Kafka, Airflow, Spark and more. They will review the challenges they faced along the way, and the solutions that most excite them for the future.
Greg is a co-founder and CTO of Astronomer. Astronomer's mission is to make it simple for anyone to leverage the power of Apache Airflow. Previously Greg has co-founded several other technology startups and a software and design consultancy.