大規模分散システムの現在 -- Twitter
↓↓↓↓訂正あります。↓↓↓↓ 2018/07/02に株式会社エフコード社内で行われた勉強会のスライドです。 訂正版(随時更新中): https://docs.google.com/presentation/d/15HOMfAbtdWwO48njcB8IdkN3kVAMu3wsmZo0O3S-f_4/edit?usp=sharing 専門家による資料・専門家向けの資料ではありません。自分自身で学習し、論文・文献等を読解してまとめた内容となります。間違い等あるかもしれませんが、あれば是非コメント頂ければと思います。 【訂正事項】 スライド16: 誤：たった一つのプロセスが故障しただけでも有限時間で合意できない 正：たった一つのプロセスが故障しうるだけでも有限時間で合意できない スライド20: 誤: 重要: あるschedule σ1, σ2 がdisjoint (nodeが被ってない) なら
Systems Programming at Twitter
Systems Programming at Twitter Facebook, October 30, 2012 Marius Eriksen Twitter Inc. (Press space or enter to navigate to the next slide, left arrow to go backwards.) A history lesson Twitter evolves 2009: Pure Ruby-on-Rails app with MySQL; lots of memcache. Materialized timelines into memcaches. Social graph moved to a service. Delayed work through queues. 2010: Starting to move timelines out to
Scala.io
Finagle is Twitter's open source RPC library that allows composing asynchronous RPC requests like functions. It provides three key abstractions: Futures for asynchronous computations, Services for RPC functions, and ServiceFactories for creating Services. Finagle handles load balancing, connection pooling, failure detection, and other distributed systems concerns through composable layers above a
Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014
Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014 Apache Parquet is an open-source columnar storage format for efficient data storage and analytics. It provides efficient compression and encoding techniques that enable fast scans and queries of large datasets. Parquet 2.0 improves on these efficiencies through techniques like delta encoding, dictionary encoding, run-length
Summingbird: Streaming MapReduce at Twitter
Summingbird is a platform for streaming map/reduce used at Twitter to build aggregations in real-time or on hadoop. When the programmer describes her job, that job can be run without change on Storm or Hadoop. Additionally, summingbird can manage merging realtime/online computations with offline batches so that small errors in real-time do not accumulate. Put another way, summingbird gives eventua
Parquet Hadoop Summit 2013
Parquet is a columnar storage format for Hadoop data. It was developed by Twitter and Cloudera to optimize storage and querying of large datasets. Parquet provides more efficient compression and I/O compared to traditional row-based formats by storing data by column. Early results show a 28% reduction in storage size and up to a 114% improvement in query performance versus the original Thrift form
Scaling Big Data Mining Infrastructure Twitter Experience
The analytics platform at Twitter has experienced tremendous growth over the past few years in terms of size, complexity, number of users, and variety of use cases. In this talk, we’ll discuss the evolution of our infrastructure and the development of capabilities for data mining on “big data”. We’ll share our experiences as a case study, but make recommendations for best practices and point out o
Hadoop and Vertica: The Data Analytics Platform at Twitter
The document discusses Twitter's data analytics platform, including Hadoop and Vertica. It outlines Twitter's data flow, which ingests 400 million tweets daily into HDFS, then uses various tools like Crane, Oink, and Rasvelg to run jobs on the main Hadoop cluster before loading analytics into Vertica and MySQL for web tools and analysts. It also describes Twitter's heterogeneous technology stack a
第3回Twitter研究会公式サイト
開催の目的とゴール Twitterはコミュニケーションインフラツールとして確固たる位置を占めるに至りました。 IT企業や地方自治体はTwitterと連携することによってサービスを如何に向上させるかが 大きな課題となってきています。また１ユーザのたった1つの「つぶやき」が大きな 社会問題となり、ひいては世界を動かす原動力になる場合も多数見られるようになってきました。 この複雑に絡み合う「つぶやき」がどのような挙動を起こすのか、分析をする動きが 活発になってきています。 本研究会は、マーケティングから技術的観点も含めてTwitterを見つめなおすことを 目的とします。そのため、講演テーマについては敢えて制約をしません。 また、参加者についても多様なバックグラウンドを持った方を想定しています。ゆえに 参加者の想定としてTwitterに関わるあらゆるステークホルダーを指します。例えば 「ユーザ」「
Hadoop at Twitter (Hadoop Summit 2010)
Kevin Weil presented on Hadoop at Twitter. He discussed Twitter's data lifecycle including data input via Scribe and Crane, storage in HDFS and HBase, analysis using Pig and Oink, and data products like Birdbrain. He described how tools like Scribe, Crane, Elephant Bird, Pig, and HBase were developed and used at Twitter to handle large volumes of log and tabular data at petabyte scale.Read less
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Large Scale EventLog Management @Twitter
Presto at Twitter
Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twitter (Alex Levenson)
Data Platform at Twitter: Enabling Real-time & Batch Analytics at Scale
Using Visualizations to Monitor Changes and Harvest Insights from a Global-scale Logging Infrastructure at Twitter
Spark at Twitter - Seattle Spark Meetup, April 2014
A Birds-Eye View of Pig and Scalding Jobs with hRaven
Parquet Twitter Seattle open house
Twitter Big Data
Everything I Ever Learned About JVM Performance Tuning @Twitter
