Data Council Blog

Data Council Blog

Community, Metadata Management, and More: Top 10 Links From Across the Web

Here's our March 2021 roundup of links from across the web that we selected for you:

1. How to Build a Community (Fishtown Analytics)

Claire Carroll's first personal blog post on community-building is a must-read. As Fishtown Analytics' community manager for the last 2.5 years, she's arguably behind the success of the dbt community and its best-in-class practices, so we expected good advice… but she really hit the ball out of the park with this one! The key takeaway is that you should start with wondering 'why' you want to build a community. Make sure to read the full post to understand why it received so much praise.
 

Open Source Highlight: PostHog

PostHog provides open-source product analytics, which users can deploy on their own infrastructure to collect every event on their website or app without having to send the data to third parties - an increasing source of concern in times of GDPR and CCPA.

dbt at Shopify, Active Learning, and More: Top 10 Links From Across the Web

Here's our February 2021 roundup of links from across the web that we picked for you:

1. dbt at Shopify (Data Engineering Podcast)

The Data Engineering Podcast recently featured a very interesting discussion about dbt at Shopify. Engineering manager Zeeshan Qureshi and senior data engineer Michelle Ark explained how dbt answered Shopify’s need for an SQL-based solution that its data scientists could use autonomously. They also mentioned some of the best practices they followed for staging, and cost considerations related to BigQuery. Last but not least, they touched on some extensions they are considering, such as implementing Great Expectations for data quality control.

Open Source Highlight: OpenLineage

OpenLineage is an API for collecting data lineage and metadata at runtime. While initiated by Datakin, the company behind Marquez, it was developed with the aim to create an open standard. As Datakin’s CTO Julien Le Dem explained in a blog post announcing the launch, OpenLineage is meant to answer the industry-wide need for data lineage, while making sure efforts in that direction aren’t fragmented or duplicated.

Storing Cold Metadata, Snowflake Data Cloud, and More: Top 10 Links From Across the Web

Here's our January 2021 roundup of links from across the web that could be relevant to you:

1. Storing Cold Metadata with Alki (Dropbox)

Dropbox shared insights into Alki, the petabyte-scale metadata store it designed for infrequently accessed metadata (“cold data”). The post details how one-size-fits-all database Edgestore was reaching capacity limits, and why audit logs were a good candidate to be moved elsewhere than on costly SSDs. After considering off-the-shelf options, the team settled on building its own solution on top of AWS services: Alki; with DynamoDB as the hot store, and S3 as the cold store. Like HBase or Cassandra, Alki is based on log-structured merge-trees (LSM trees), but is better suited to handle hot-then-cold audit logs, as well as future use cases at Dropbox.

The Modern Data Stack, Metadata Architectures, and More: Top 10 Links From Across the Web

Here's our December 2020 roundup of links from across the web that could be relevant to you:

1. The Modern Data Stack (Fishtown Analytics)

This long-form post on the dbt blog is a must-read. Titled “The Modern Data Stack: Past, Present, and Future,” it answers the question that Tristan Handy has been asking himself for the past two years: “What happened to the massive innovation we saw from 2012-2016?” His carefully thought-out analysis covers the natural cycles of technological shifts, defines the phase we are in as a ‘deployment’ one, and points out high-impact opportunity areas for the next few years - which you might find particularly useful if you are considering launching a new product.

NLP Heroes, Pinot, Data Testing, and More: Top 10 Links From Across the Web

Here's our November 2020 roundup of good reads and podcast episodes that might be relevant for your career in data:

1. Heroes of NLP: Quoc Le (Deeplearning.ai)

NLP researcher Quoc Le was recently Andrew Ng’s guest as part of the ‘Heroes of NLP’ video series. Their discussion covered Le’s impressive journey, from growing up in Vietnam and developing his first basic chatbot in high school to becoming Google Brain’s first intern, and everything that followed. This includes the ‘Google Cat’ experiment, the Meena chatbot project, and work on Seq2Seq models. Check out the conversation here, and consider subscribing to the series to hear from other guests such as Chris Manning, Kathleen McKeown, and Oren Etzioni.

State of AI, Data Quality, and More: Top 10 Links From Across the Web

Here's our October 2020 roundup of good reads and podcast episodes that might be relevant to you as a data professional:

1. Multiplayer Editing: a Pragmatic Approach (Hex)

Data collaboration startup Hex published a great long read on its approach to live collaboration . Written by software engineer Mac Lockard, it takes a look at the respective pros and cons of Operational Transforms and Conflict-free Replicated Data Types (CRDTs), before explaining the solution that Hex adopted. Inspired by Figma's hybrid approach, it can also be described as "Atomic Operations (AO), as all edits to application state are broken down to their smallest atomic parts." "If the application you are building can rely on last-writer-wins semantics, Atomic Operations might provide a more pragmatic approach," the post concludes. This is a highly recommended read if you are pondering about a similar decision.  

Hot Data Tools pt. 2, End-to-End Data Scientists, and More: Top 10 Links From Across the Web

Here's our September 2020 roundup of good reads and podcast episodes that might be relevant to you as a data professional:

1. What Data Tools Don't Do (Data Council)

Our founder Pete Soderling co-authored a follow-on piece to his previous post with Great Expectations' core contributor Abe Gong and Partner at Amplify Partners Sarah Catanzaro, for which they had interviewed the makers of some of the hottest data tools. The focus is still the same: rather than what their data tools can do, we hear about what they don't do, as a way to better understand how they fit together. From ApertureData to Xplenty, this new installment covers 21 new tools, and you can read it here.

Large Datasets, Are Dashboards Dead, and More: Top 10 Links From Across the Web

Here's our August 2020 roundup of good reads and great podcast episodes for anyone working with data:

1. Processing Large Datasets with Python

AI engineer and author J.T. Wolohan was recently a guest of the Heroku’s Code[ish] podcast to discuss the contents of his book, “Mastering Large Datasets with Python.” Listen to the episode here or read the transcript for some practical advice on using Python to deal with massive datasets, especially in the context of machine learning.