Real-time MapReduce: Why MapReduce is making a comeback
What if we could adapt MapReduce to real-time data processing? Spoiler: it’s awesome and we’re building a next-gen data platform based on it! In the 2020’s, MapReduce isn’t a processing model that will turn many heads. It’s been around for well over a decade, and is easy to write off as an idea that might’ve been groundbreaking in its time, but is dated today. MapReduce was a big deal for a reason
Get Started with TensorFlow Transform | TFX
This guide introduces the basic concepts of tf.Transform and how to use them. It will: Define a preprocessing function, a logical description of the pipeline that transforms the raw data into the data used to train a machine learning model. Show the Apache Beam implementation used to transform data by converting the preprocessing function into a Beam pipeline. Show additional usage examples. Setup
Dataflow overview | Google Cloud
Send feedback Dataflow overview Stay organized with collections Save and categorize content based on your preferences. Dataflow is a Google Cloud service that provides unified stream and batch data processing at scale. Use Dataflow to create data pipelines that read from one or more sources, transform the data, and write the data to a destination. Typical use cases for Dataflow include the followi
DataflowTemplates/v2/cdc-parent/README.md at main · GoogleCloudPlatform/DataflowTemplates
This directory contains components for a Change-data Capture (CDC) solution to capture data from an MySQL database, and sync it into BigQuery. The solution relies on Cloud Dataflow, and Debezium, an excellent open source project for change data capture. To implement the CDC solution in this repository: Deploy a Debezium embedded connector for MySQL Start a Dataflow pipeline that syncs MySQL and Bi
メルペイにおける Dataflow Template の活用 | メルカリエンジニアリング
この記事は MERPAY TECH OPENNESS MONTH の9日目の記事です。 株式会社メルペイのSolutionチームでデータエンジニアをやっている @orfeon です。 この記事ではGCPのデータストアサービスやプロジェクトをまたいでデータを活用するためにメルペイで活用しているDataflow Templateについて紹介します。 メルペイでは各マイクロサービスで Cloud Spanner、BigQuery、Cloud Storage、Cloud SQL、Cloud Datastore など様々なGCPのデータストアサービスを活用しています。 マイクロサービスによってはこれらデータストアサービス間でデータを加工・移動して活用しなければならないこともあります。 例としては、Spannerから店舗ごとに売上を集計してCSVファイルとしてCloud Storageに保存してダウン
Cloud Composer & Dataflow によるバッチETLの再構築 #data_ml_engineering / 20190719
データとML周辺エンジニアリングを考える会#2の発表資料です。 https://data-engineering.connpass.com/event/136756/
kati について - 兼雑記
https://github.com/google/kati kati について、ドキュメント書こう…と思っていたのですがなかなか進まないので、とりあえず日本語で書いてみることにしました。何書くかがあまり明確じゃないテーマなので、何書くか考えるのと英語考えるのを両方同時にやるのが少し大変で。 動機 kati は GNU make のクローンです。いずれ完全なコンパチになると嬉しいですが、なかなか難しいだろうと個人的には諦めています。用途に対して実用的ならば良いかなと。 動機としては、 Android platform のビルドシステムが、なかなかシュールな GNU make 黒魔術で構成されていて、 make が実際になんかしはじめるまでが遅かったので、そこを高速化したいというものでした。 ビルドシステムが遅いという時、まずだいたいヌルビルドとフルビルドの2点を考えます。ヌルビルドてのは生
Guide to common Cloud Dataflow use-case patterns, Part 1 | Google Cloud Blog
Turning data into business value reportHarvard Business Review Analytic Services examines how leaders successfully drive results from their enterprise data strategy. DOWNLOAD Editor’s note: This is part one of a series on common Dataflow use-case patterns. You can find part two here. ">Editor’s note: This is part one of a series on common Dataflow use-case patterns. You can find part two her
How to handle mutating JSON schemas in a streaming pipeline | Google Cloud Blog
Forrester Wave: Cloud Data Warehouse, Q1 2021 ReportLearn why Forrester gave BigQuery a 5 out of 5 score across 19 different criteria. DOWNLOAD Back in May, Paul Barnes (Google Customer Engineer) and I headed over to Square Enix’s offices to meet up with Chris Windsor, Square Enix’s Data Engineering Lead, for a streaming architecture jam session. ">Back in May, Paul Barnes (Google Cust
Christopher Olston - Publications
Papers by Christopher Olston Research community reports Machine learning Enterprise data management Continuous data processing workflows Data pipeline programming & debugging Large-scale data processing Web monitoring & crawling Web search Web application scalability Distributed data monitoring Data visualization Research community reports Numerous co-authors. The Beckman Report on Database Resear
Red Book, 5th ed. Ch. 5: Large-Scale Dataflow Engines
Of the many developments in data management over the past decade, MapReduce and subsequent large-scale data processing systems have been among the most disruptive and the most controversial. Cheap commodity storage and rising data volumes led many Internet service vendors to discard conventional database systems and data warehouses and build custom, home-grown engines instead. Google’s string of p
TensorBoard: Graph Visualization | TensorFlow
Overview TensorBoard’s Graphs dashboard is a powerful tool for examining your TensorFlow model. You can quickly view a conceptual graph of your model’s structure and ensure it matches your intended design. You can also view a op-level graph to understand how TensorFlow understands your program. Examining the op-level graph can give you insight as to how to change your model. For example, you can r
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
How to build an end-to-end propensity to purchase solution using BigQuery ML and Kubeflow Pipelines
Data analytics design patterns | Cloud Architecture Center | Google Cloud
Redirecting
The right metrics to monitor cloud data pipelines | Google Cloud Blog
GitOps: VPC内リソースにアクセスするDataflowジョブのデプロイをCloud BuildとGKEで自動化する
Use Dataflow SQL | Google Cloud
Data Integration Platform for Enterprise Companies | StreamSets
http://idl.cs.washington.edu/files/2015-ReactiveVega-InfoVis.pdf