Using logs to build a solid data infrastructure (or: why dual writes are a bad idea) | Confluent
This is an edited transcript of a talk I gave at the Craft Conference 2015. The video and slides are also available. How does your database store data on disk reliably? It uses a log. How does one database replica synchronise with another replica? It uses a log. How does a distributed algorithm like Raft achieve consensus? It uses a log. How does activity data get recorded in a system like Apache
NimbusSanL-Regu
The Unified Logging Infrastructure for Data Analytics at Twitter George Lee, Jimmy Lin, Chuang Liu, Andrew Lorek, and Dmitriy Ryaboy Twitter, Inc. @GeorgeJLee @lintool @chuangl4 @mrtall @squarecog ABSTRACT In recent years, there has been a substantial amount of work on large-scale data analytics using Hadoop-based platforms running on large clusters of commodity machines. A less- explored topic is
Hadoop and Vertica: The Data Analytics Platform at Twitter
The document discusses Twitter's data analytics platform, including Hadoop and Vertica. It outlines Twitter's data flow, which ingests 400 million tweets daily into HDFS, then uses various tools like Crane, Oink, and Rasvelg to run jobs on the main Hadoop cluster before loading analytics into Vertica and MySQL for web tools and analysts. It also describes Twitter's heterogeneous technology stack a
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
処理を実行中です
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Scaling Big Data Mining Infrastructure: The Twitter Experience Jimmy Lin and Dmitriy Ryaboy Twitter, Inc. @lintool @squarecog ABSTRACT The analytics platform at Twitter has experienced tremen- dous growth over the past few years in terms of size, com- pl
