Data Catalogs Are Dead; Long Live Data Discovery
Image courtesy of Andrey_Kuzmin on ShutterstockAs companies increasingly leverage data to power digital products, drive decision making, and fuel innovation, understanding the health and reliability of these most critical assets is fundamental. For decades, organizations have relied on data catalogs to power data governance. But is that enough? Debashis Saha, VP of Engineering at AppZen, formerly
データ基盤のメタデータを継続的に管理できる仕組みを作る - Hatena Developer Blog
こんにちは。MackerelチームでCRE(Customer Reliability Engineer)をしているid:syou6162です。 CREチームではカスタマーサクセスを進めるため、最近データ分析により力を入れています(参考1, 参考2)。データ分析を正確に行なうためには、データに関する正確な知識が必要です。今回はより正確なデータ分析を支えるためのメタデータを継続的に管理する仕組みについて書いてみます。 データに対する知識: メタデータ データ分析を正確に行なうためには、データ自身に関する知識(=メタデータ)が必要です。例えば、Mackerelのデータ分析タスクでは以下のような知識が必要とされることが多いです。 このテーブル / カラムは何のためのテーブルなのか 似たようなカラムとの違い 集計条件の違い、など データがどのような値を取り得るか SELECT column, COU
How We Improved Data Discovery for Data Scientists at Spotify - Spotify Engineering
How We Improved Data Discovery for Data Scientists at Spotify At Spotify, we believe strongly in data-informed decision making. Whether we’re considering a big shift in our product strategy or we’re making a relatively quick decision about which track to add to one of our editorially-programmed playlists, data provides a foundation for sound decision making. An insight is a conclusion drawn from d
How LinkedIn, Uber, Lyft, Airbnb and Netflix are Solving Data Management and Discovery for Machine Learning Solutions
How LinkedIn, Uber, Lyft, Airbnb and Netflix are Solving Data Management and Discovery for Machine Learning Solutions When comes to machine learning, data is certainly the new oil. The processes for managing the lifecycle of datasets are some of the most challenging elements of large scale machine learning solutions. Data ingestion, indexing, search, annotation, discovery are some of the aspects r
Open Sourcing WhereHows: A Data Discovery and Lineage Portal
In modern data-driven businesses, the complexity that arises from fast-paced analytics, data mining and ETL processes makes metadata increasingly important. In this blog post, we share our own journey and a new open source effort that aims to boost productivity and data provenance. WhereHows, a project of the LinkedIn Data team, works by creating a central repository and portal for the processes,
Overview
Collect, aggregate, and visualize a data ecosystem's metadata View on GitHub Quickstart Download Overview Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem’s metadata. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of da
Databook: Turning Big Data into Knowledge with Metadata at Uber
You’re seeing information for Japan . To see local features and services for another location, select a different city. Show more From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data. Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our
Metacat: Making Big Data Discoverable and Meaningful at Netflix
by Ajoy Majumdar, Zhen Li Most large companies have numerous data sources with different data formats and large data volumes. These data stores are accessed and analyzed by many people throughout the enterprise. At Netflix, our data warehouse consists of a large number of data sets stored in Amazon S3 (via Hive), Druid, Elasticsearch, Redshift, Snowflake and MySql. Our platform supports Spark, Pre
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
The Rise of the Metadata Lake
The Government Data Quality Framework
https://hpi.de/fileadmin/user_upload/fachgebiete/naumann/publications/2017/SIGMOD_2017_Tutorial_Data_Profiling.pdf
egeria
GitHub - odpi/egeria: Egeria core
Business Glossary support in Google Data Catalog
Joshua Shinavier – The Knowledge Graph Conference
Building an Enterprise Knowledge Graph at Uber: Lessons from Reality
データテクノロジースペシャル：Yahoo! JAPANにおけるメタデータ管理の試み
7 Things Every Data Worker Should Know About Metadata
Google Dataset Search: Building a search engine for datasets in an open Web ecosystem
Marquez: A Metadata Service for Data Abstraction, Data Lineage, and Event-based Triggers