Timely (and Stateful) Processing with Apache Beam
In a prior blog post, I introduced the basics of stateful processing in Apache Beam, focusing on the addition of state to per-element processing. So-called timely processing complements stateful processing in Beam by letting you set timers to request a (stateful) callback at some point in the future. What can you do with timers in Beam? Here are some examples: You can output data buffered in state
Stateful processing with Apache Beam
Beam lets you process unbounded, out-of-order, global-scale data with portable high-level pipelines. Stateful processing is a new feature of the Beam model that expands the capabilities of Beam, unlocking new use cases and new efficiencies. In this post, I will guide you through stateful processing in Beam: how it works, how it fits in with the other features of the Beam model, what you might use
Big Data Processing at Spotify: The Road to Scio (Part 2) - Spotify Engineering
In this part we’ll take a closer look at Scio, including basic concepts, its unique features, and concrete use cases here at Spotify. Basic Concepts Scio is a Scala API for Apache Beam and Google Cloud Dataflow. It was designed as a thin wrapper on top of Beam’s Java SDK, while offering an easy way to build data pipelines in idiomatic Scala style. We drew most of our inspiration for the API from S
Google Cloud Dataflow でアクセスログを監視して Google Cloud Functions で Slack へアラートを送信するの巻 - おくみん公式ブログ
『Google Cloud Dataflow で Google BigQuery へストリーミング ETL するの巻』で加工したアクセスログを集計し、一定の条件を満たすと Slack へアラートを飛ばすシステムを作りました。 Apache Beam(Scio) + Google Cloud Dataflow を用いてログの集計と監視を行い、問題のあるアクセスが見つかったら Google Cloud Functions で Slack へ通知するという構成です。 ストリーミング ETL の記事と同様にその仕組みを紹介します。 アーキテクチャ 次の流れで監視を行います。 Google Cloud Dataflow で Google Cloud Pub/Sub のトピック「okumin-com-access-log-refined-v1」から取得したアクセスログを解析し、問題があれば「okumi
Google Cloud Dataflow で Google BigQuery へストリーミング ETL するの巻 - おくみん公式ブログ
okumin.com のアクセスログを Google BigQuery で分析するために、ETL パイプラインを構築しました。 Apache Beam(Scio) + Google Cloud Dataflow を用いてログの加工及び BigQuery へのストリーミングインサートを行うという構成です。 この記事ではその全体像と個々のコンポーネントの簡単な説明を行います。 アーキテクチャ 次の流れで ETL を行っています。 Google Container Engine(GKE) 上に配置した Nginx コンテナのログを Stackdriver Logging へ集約 Stackdriver Logging のログを Google Cloud Pub/Sub のトピック「okumin-com-access-log-raw-v1」へエクスポート Google Cloud Dataflow
Watermarks, Tables, Event Time, and the Dataflow Model | Confluent
The Google Dataflow team has done a fantastic job in evangelizing their model of handling time for stream processing. Their key observation is that in most cases you can’t globally order data arrival, which means that stream processing must handle out-of-order data. The Dataflow model of computation has integrated a system for coping with this into the Beam API. This has been widely copied by diff
Apache Beam publishes the first stable release
The Apache Beam community is pleased to announce the availability of version 2.0.0. This is the first stable release of Apache Beam, signifying a statement from the community that it intends to maintain API stability with all releases for the foreseeable future, and making Beam suitable for enterprise deployment. This first stable release is the third important milestone for the Apache Beam commun
Apache Beam: A unified model for batch and stream processing data
Google Cloud Dataflow is a fully managed cloud service for batch and streaming data processing. It provides a unified programming model and DSL that can process both batch and streaming data on platforms like Spark, Flink, and Google's managed Dataflow service. Dataflow uses concepts like pipelines, PCollections, transforms, and windowing to process data according to event time while controlling l
Using Cloud Dataflow to run parallel predictions with your TensorFlow model - めもめも
Suppose that you've finished training your prediction model with TensorFlow, yeay! Now you have to make predictions with the trained model for tens of thousands of data. How will you do it? I will show you one of possible choices. You can use Cloud Dataflow for general parallel batch processing and it's not hard to apply machine learning models in this framework. What is Cloud Dataflow? Cloud Data
Apache Gearpump next-gen streaming engine
Our talk on Apache Gearpump at Strata+Hadoop World Beijing 2016.
Unbounded bounded-data-strangeloop-2016-monal-daxini
The need for gleaning answers from data in real-time is moving from nicety to a necessity. There are few options to analyze the never-ending stream of unbounded data at scale. Let’s compare and contrast the core principles and technologies the different open source solutions available to help with this endeavor, and where in the future processing engines need to evolve to solve processing needs at
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Powerful and modular IO connectors with Splittable DoFn in Apache Beam
Realizing the promise of portability with Apache Beam
beam/runners at master · apache/beam
http://www.jesse-anderson.com/2016/12/beams-pico-wordcount/
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Maximilian Michels - Flink and Beam
Rise of Intermediate APIs - Beam and Alluxio at Alluxio Meetup 2016
Netflix Keystone—Cloud scale event processing pipeline
Apache Beam: The Case for Unifying Streaming APIs