第16回 並列データ処理系 Apache Tez | gihyo.jp
はじめに 今回は、Apache Hadoop上で動作する並列データ処理系Apache Tezについて解説します。 MapReduceの制約 本連載第13回で述べたように、Hadoop MapReduceは、MapとReduceからなる単純なインタフェースを有し、多くのデータ処理を記述できる汎用的な並列データ処理系（フレームワーク）である反面、その単純さによりいくつかの性能的な課題が存在すると考えられます。 たとえば、複雑なジョブをMapReduceで実行する場合、MapとReduceからなるMapReduceジョブを複数段連ねて実行する必要があります（図1）。当該ケースにおいては、MapReduceジョブの間において、分散ファイルシステムを介したデータの入出力が行われてしまい、当該入出力は、性能の観点においてはオーバーヘッドであるため、ジョブの実行時間を長くする原因の1つとなりえます。
Apache Tez : Accelerating Hadoop Query Processing
Apache Tez is the new data processing framework in the Hadoop ecosystem. It runs on top of YARN - the new compute platform for Hadoop 2. Learn how Tez is built from the ground up to tackle a broad spectrum of data processing scenarios in Hadoop/BigData - ranging from interactive query processing to complex batch processing. With a high degree of automation built-in, and support for extensive custo
Cloudera Blog
Enterprises see embracing AI as a strategic imperative that will enable them to stay relevant in increasingly competitive markets. However, it remains difficult to quickly build these capabilities given the challenges with finding readily available talent and resources to get started rapidly on the AI journey. Cloudera recently signed a strategic collaboration agreement with Amazon […] Read blog p
Apache Tez: Accelerating Hadoop Query Processing
This document discusses Apache Tez, a framework for accelerating Hadoop query processing. Some key points: - Tez is a dataflow framework that expresses computations as directed acyclic graphs (DAGs) of tasks, allowing for optimizations like container reuse and locality-aware scheduling. - It is built on YARN and provides a customizable execution engine as well as APIs for applications like Hive an
Cloudera Blog
Riding the wave of the generative AI revolution, third party large language model (LLM) services like ChatGPT and Bard have swiftly emerged as the talk of the town, converting AI skeptics to evangelists and transforming the way we interact with technology. For proof of this megatrend look no further than the instant success of ChatGPT, […] Read blog post
TezProposal - INCUBATOR - Apache Software Foundation
Tez Abstract Tez is an effort to develop a generic application framework which can be used to process arbitrarily complex data-processing tasks and also a re-usable set of data-processing primitives which can be used by other projects. Proposal Tez is a proposal to develop a generic application which can be used to process complex data-processing task DAGs and runs natively on Apache Hadoop YARN.
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Apache Tez – Present and Future
GitHub - apache/tez: Apache Tez
GitHub - apache/incubator-tez: Mirror of Apache Tez (Incubating)