Metabase linkedin professional#

This in turn would have allowed us to focus on onboarding and evolving strongly opinionated metadata models without worrying about the lower layers of the stack. It would have been more scalable had we designed a general architecture that is agnostic to the metadata model it stores and serves. A small change to the metadata model will lead to a cascade of changes required up and down the stack. This results in an opinionated API, data model, and storage format. General is better than specific: WhereHows is strongly opinionated about how the metadata for a dataset or a job should look like.This push-based approach also ensures a more timely reflection of new and updated metadata.

It is more scalable to have individual metadata providers push the information to the central repository via APIs or messages. Push is better than pull: While pulling metadata directly from the source seems like the most straightforward way to gather metadata, developing and maintaining a centralized fleet of domain-specific crawlers quickly becomes a nightmare.Here is a summary of the lessons we learned from scaling WhereHows: However, we came to realize WhereHows had fundamental limitations that prevented it from meeting our evolving metadata needs. At LinkedIn, we have also been busy expanding our scope of metadata collection to power new use cases while preserving fairness, privacy, and transparency. For example, tools developed in this space include AirBnb’s Dataportal, Uber’s Databook, Netflix’s Metacat, Lyft’s Amundsen, and most recently Google’s Data Catalog. Since our initial release of WhereHows in 2016, there has been a growing interest in the industry to improve the productivity of data scientists by using metadata. WhereHows also featured a search engine to help locate the datasets of interest. The type of metadata stored includes both technical metadata (e.g., location, schema, partitions, ownership) and process metadata (e.g., lineage, job execution, lifecycle information). To increase the productivity of LinkedIn’s data team, we had previously developed and open sourced WhereHows, a central metadata repository and portal for datasets.

To help us continue scaling productivity and innovation in data alongside this growth, we created a generalized metadata search and discovery tool, DataHub. As the data grows in volume and richness, it becomes increasingly challenging for data scientists and engineers to discover the data assets available, understand their provenances, and take appropriate actions based on the insights.

Metabase linkedin professional#

You can read more on the journey of open sourcing the platform here.Īs the operator of the world’s largest professional network and the Economic Graph, LinkedIn’s Data team is constantly working on scaling its infrastructure to meet the demands of our ever-growing big data ecosystem. Co-authors: Mars Lan, Seyi Adebajo, Shirshanka DasĮditor’s note: Since publishing this blog post, the team open sourced DataHub in February 2020.