Distributed Stream Processing on Fluentd / #fluentd

This document discusses NHN Japan's use of Fluentd for log collection, conversion, and analysis. It summarizes their architecture which involves collecting logs from servers using Scribeline, delivering them to Fluentd processes, converting the logs to a structured format using Fluentd plugins, writing the structured data to HDFS, and performing analysis and reporting with Hive. Key aspects discus

[Jxck]

やっぱ、 tgmr 先生スゲーなぁ。。

[mooz]

127 のサーバから 146 のログストリーム．70,000 メッセージ/秒, 120 Mbps (peak). 89 fluentd インスタンスを 12 ノードで．

[antipop]

これはすごい

[riywo]

すごい

[kzk]

by @tagomoris at #fluentd meetup