[B! GearPump][stream] kimutanskのブックマーク

Storm over gearpump

kimutansk 2017/02/01

確かに書いてみるとこれモダンなStormやんという印象はあります。そういう共通点があるからこそそのまんま乗せやすいわけですか。

リンク

Functional Comparison and Performance Evaluation of Streaming Frameworks

4. Apache Spark Streaming* Aapche Flink* Apache Storm* Apache Storm Trident* Apache Gearpump* Twitter Heron* This is the critical part, as it affects many features Micro-Batch Checkpoint per Batch Continuous Streaming Checkpoint “per Batch” Source Operator Sink Acker Source Operator Sink JobManager/ HDFS id offset state str ack Source Operator Sink Driver Storage Storage job status HDFS id offset

kimutansk 2016/10/12

各ストリーム処理基盤同士の比較資料。Gearpumpは比較的基盤部が薄いので性能は全般的に高いが、アプリを実現する際の機能としてFlink等に劣り、開発にも手間が大きいという傾向？

リンク

Gearpump akka streams

The document discusses integrating Akka streams with the Gearpump big data streaming platform. It provides background on Akka streams and Gearpump, and describes how Gearpump implements a GearpumpMaterializer to rewrite the Akka streams module tree for distributed execution across a Gearpump cluster. Key points covered include the object models of Akka streams and Gearpump, prerequisites for big d

kimutansk 2016/10/09

ようやく共通のDSLで透過的に起動可能と。あとはModuleをより簡易に書ける方が・・・Akka系の常か、カスタマイズ性は高いが開発効率は微妙なわけで。ただこれはトレードオフか。

リンク

Apache Gearpump next-gen streaming engine

Gearpump is an Apache Incubator project that provides a real-time streaming engine based on Akka. It allows users to easily program streaming applications as a directed acyclic graph (DAG) and handles issues like out-of-order data processing, flow control, and exactly-once processing to prevent lost or duplicated states. Gearpump also enables dynamic updates to the DAG at runtime and provides visu