Back at my old job in ~2016, we built a cheap homegrown data warehouse via Postg... | Hacker News
Back at my old job in ~2016, we built a cheap homegrown data warehouse via Postgres, SQLite and Lambda.Basically, it worked like this: - All of our data lived in compressed SQLite DBs on S3. - Upon receiving a query, Postgres would use a custom foreign data wrapper we built. - This FDW would forward the query to a web service. - This web service would start one lambda per SQLite file. Each lambda
Exploring Notion's Data Model: A Block-Based Architecture | Notion
A generation of pioneers (Doug Engelbart, Ted Nelson, Alan Kay, and many more) saw the computer as tool to augment human problem-solving by giving people power over information. Today, that information mostly remains siloed across tools. Take cloud-based document editors, where pages are their smallest atomic unit. Information is locked inside of pages and files and folders — that’s reminiscent of
データ基盤は「データで」改善する − 中間テーブル作成の事例｜Mercari Analytics Blog
メルカリAnalytics Infraチームの@__hiza__です。 この記事では、メルカリにおけるデータ基盤の整備について紹介します。 今回は、膨大な生のテーブルについてどのテーブルから中間テーブル化すると効果的か、データにもとづいて優先順位を付けた事例を説明します。 また、あわせて大規模なデータ基盤を改善する際に「データ基盤の利用状況のログ」が役立つことをお話したいと思います。 中間テーブルを作る意義データ分析用の中間テーブルを作成する意義を簡単におさらいします。 例えば、WebサービスのRDBに入っているデータで分析を行う場合に以下のような加工をしたテーブルを用意すると分析が便利になります。 例) 生テーブルを分析に使いやすくする加工の例 よくjoinして利用する複数のテーブルをあらかじめjoinしておく コード化された値を人が見て分かる値に変換しておく(都道府県コード01→北海道
Project RADAR: Intelligent Early Fraud Detection System with Humans in the Loop
You’re seeing information for Japan . To see local features and services for another location, select a different city. Show more Introduction Uber is a worldwide marketplace of services, processing thousands of monetary transactions every second. As a marketplace, Uber takes on all of the risks associated with payment processing. Uber partners who use the marketplace to provide services are paid
GitHub - fatiando/pooch: A friend to fetch your data files
Just want to download a file without messing with requests and urllib? Trying to add sample datasets to your Python package? Pooch is here to help! Pooch is a Python library that can manage data by downloading files from a server (only when needed) and storing them locally in a data cache (a folder on your computer). Pure Python and minimal dependencies. Download files over HTTP, FTP, and from dat
The Building Blocks of a Modern Data Platform
Photo by Adam Nowakowski on UnsplashIf you Google “modern data platform”, you’ll immediately be bombarded with advertisements and lots of companies professing that they are the one true data platform. Not so helpful, right? So what the heck is a modern data platform? What does that even mean, and what does it look like in 2021? The short answer: a modern data platform is a collection of tools and
Nemo: Data discovery at Facebook
Large-scale companies serve millions or even billions of people who depend on the services these companies provide for their everyday needs. To keep these services running and delivering meaningful experiences, the teams behind them need to find the most relevant and accurate information quickly so that they can make informed decisions and take action. Finding the right information can be hard for
How to analyse 100s of GBs of data on your laptop with Python
Many organizations are trying to gather and utilise as much data as possible to improve on how they run their business, increase revenue, or how they impact the world around them. Therefore it is becoming increasingly common for data scientists to face 50GB or even 500GB sized datasets. Now, these kind of datasets are a bit… uncomfortable to use. They are small enough to fit into the hard-drive of
dataset 1.6.2 documentation
Although managing data in relational databases has plenty of benefits, they’re rarely used in day-to-day work with small to medium scale datasets. But why is that? Why do we see an awful lot of data stored in static files in CSV or JSON format, even though they are hard to query and update incrementally? The answer is that programmers are lazy, and thus they tend to prefer the easiest solution the
DBLog: A Generic Change-Data-Capture Framework
Andreas Andreakis, Ioannis Papapanagiotou OverviewChange-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional
Presto Infrastructure at Lyft
OverviewEarly in 2017 we started exploring Presto for OLAP use cases and we realized the potential of this amazing query engine. It started as an adhoc querying tool for data engineers and analysts to run SQL in a faster way to prototype their queries, when compared to Apache Hive. A lot of internal dashboards were powered by AWS-Redshift back then and it had data storage and compute coupled toget
国会議員白書――政治家の仕事を記録・公開するウェブサイトです
【お知らせ】現在、衆議員のデータを更新中です。リンク切れや見辛い個所などありますが、ご了承ください。今回の更新から議員立法のデータが新たに加わりました。また、各種データの順位表示も始めました。（2024/01/19） ご案内 国会議員白書は、衆参両院議員の国会での活動や選挙結果などの情報をまとめたウェブサイトです。 戦後新憲法下、1947年から衆議院議員は2023年12月まで、参議院議員は2022年12月までに当選した全議員について、選挙区と選挙結果、本会議や委員会での質問・答弁・演説などの発言の一覧と統計、出席した委員会の数や種類、質問主意書の数と内容、会派や役職就任状況など情報を整理して公表しています。見えにくい国会議員の普段の活動を可視化するのが当サイトの中心的な狙いです。 最新のデータは、衆議院は49期（2021年の第49回衆院選で当選した議員）、参議院は26期（2022年の第26
Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…
By: Di Lin, Girish Lingappa, Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question — “Can I run a check myself to understand what data is behind this metric?” Now, imagine yourself in the role of a software engineer responsible for a micro-service which publishes dat
Data Engineering at SmartNews
2019年3月8日のMachine Learning Casual Talks #9で話した資料です https://mlct.connpass.com/event/120451/
早わかりSpring Data JDBC / jsug-2019-01
JSUG勉強会 2019その1 Spring Data JDBC正式リリース記念！データアクセス特集 https://jsug.doorkeeper.jp/events/86027
cURL as DSL — cURL as DSL 1.0 documentation
cURL as DSL¶ github repository Target: Curl Command: Generate Code Reset Result code is printed here. Usage¶ Select target envirionment and type curl command in above text box. You get source code that works as same as curl command. Enjoy! Supported Options¶ It doens’t support fully options of cURL. It supports only options for http 1.x in the following list: [supported options:] --basic Use HTTP
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Upgrading Data Warehouse Infrastructure at Airbnb
Jsonnet - Jsonnet Configuration Language
Netflix - データ重視のシリコンバレー文化と人間関係重視のハリウッド文化の衝突
Material Design
