MapReduce: A major step backwards - The Database Column
These search terms are highlighted: search These terms only appear in links pointing to this page: hl en&safe off&q [Note: Although the system attributes this post to a single author, it was written by David J. DeWitt and Michael Stonebraker] On January 8, a Database Column reader asked for our views on new distributed database research efforts, and we'll begin here with our views on MapReduce. Th
The Elephant was a Trojan Horse: On the Death of Map-Reduce at Google - Paper Trail
Note: this is a personal blog post, and doesn’t reflect the views of my employers at Cloudera Map-Reduce is on its way out. But we shouldn’t measure its importance in the number of bytes it crunches, but the fundamental shift in data processing architectures it helped popularise. This morning, at their I/O Conference, Google revealed that they’re not using Map-Reduce to process data internally at
Google Dumps MapReduce in Favor of New Hyper-Scale Analytics System
Google has abandoned MapReduce, the system for running data analytics jobs spread across many servers the company developed and later open sourced, in favor of a new cloud analytics system it has built called Cloud Dataflow. MapReduce has been a highly popular infrastructure and programming model for doing parallelized distributed computing on server clusters. It is the basis of Apache Hadoop, the
TezProposal - INCUBATOR - Apache Software Foundation
Tez Abstract Tez is an effort to develop a generic application framework which can be used to process arbitrarily complex data-processing tasks and also a re-usable set of data-processing primitives which can be used by other projects. Proposal Tez is a proposal to develop a generic application which can be used to process complex data-processing task DAGs and runs natively on Apache Hadoop YARN.
Welcome to YSmart's Homepage!
The homepage of YSmartOverview YSmart is a correlation aware SQL-to-MapReduce translator, which is built on top of the Hadoop platform. For a given SQL query and related table schemas, YSmart can automatically translate the query into a series of Hadoop MapReduce programs written in Java. Compared to other SQL-to-MapReduce translators, YSmart has been proved to have the following advantages: High
MapReduceは今後どうなるのか？ - 急がば回れ、選ぶなら近道
2012年の現在、割と悩んでいるのでメモっておく。 年度末ぐらいに再調査の予定。・・なので暫定ですよ。 まず前提として、現行のHadoopの実行フレームワークであるMapReduceは、実行効率は決して良くはないです。この辺が割と辛い。 とはいえ、大規模並列処理を一般的に行うという観点での品質や取り回しを考えた場合、”結果として”非常にバランスがとれており、普及している。その上で、このMapReduceですが、今後の見通しについては、潮流は今のところ二つに割れているよう見える。ので、その辺のメモ。 ■YARN 一つの方向性は、現在のHadoop2.0系で実装されているMapReduce2.0、というか、MapReduceとは別の実行基盤を利用するという方向ですね。すなわちBSPや、MPIを利用する。要は、今までの並列処理の成果をそのまま利用しましょう、という流れに近い。 MapReduce
Roundup of Big Data Forecasts and Market Estimates, 2012
From the best-known companies in enterprise software to start-ups, everyone is jumping on the big data bandwagon. The potential of big data to bring insights and intelligence into enterprises is a strong motivator, where managers are constantly looking for the competitive edge to win in their chosen markets. With so much potential to provide enterprises with enhanced analytics, insights and inte
Why the days are numbered for Hadoop as we know it – Old GigaOm
Hadoop is everywhere. For better or worse, it has become synonymous with big data. In just a few years it has gone from a fringe technology to the de facto standard. Want to be big bata or enterprise analytics or BI-compliant? You better play well with Hadoop. It’s therefore far from controversial to say that Hadoop is firmly planted in the enterprise as the big data standard and will likely rema
Mining of Massive Datasets
The book has a new Web site www.mmds.org. This page will no longer be maintained. Your browser should be automatically redirected to the new site in 10 seconds. The book has now been published by Cambridge University Press. The publisher is offering a 20% discount to anyone who buys the hardcopy Here. By agreement with the publisher, you can still download it free from this page. Cambridge Press d
MapReduce Patterns, Algorithms, and Use Cases
In this article I digested a number of MapReduce patterns and algorithms to give a systematic view of the different techniques that can be found on the web or scientific articles. Several practical case studies are also provided. All descriptions and code snippets use the standard Hadoop’s MapReduce model with Mappers, Reduces, Combiners, Partitioners, and sorting. This framework is depicted in th
Mapreduce2.0 - 急がば回れ、選ぶなら近道
次世代Hadoopの開発が進んでいる。現状の推移では、少なくとも分散クラウドでの「OSSインフラ」としてはHadoopが最有力候補であることは間違いない。クラウド上での分散処理基盤での技術競争ではGoogleやAmazonが相当抜きんでいる現在、それに対抗しうる可能性があるOSSはHadoopの潮流の延長線上にしか考えられない。その形としてHadoop-MapReduce2.0があるように見える。現在の状態で自分なりの次世代Hadoopの理解をまとめておく。基本的に全部は見切れていないので、そのあたりはあしからず。基本的に次世代Hadoopの仕組みは大きく二つの要素からなる 現在のところの柱はHDFSとMapreduce2.0の二つだ。 まずMapReduce。これは従来の「MapReduce」というものからはほど遠い。むしろ「任意」の分散処理実行フレームワークにたいして、適切なリソースを
Brown University - Data Management Group » A Comparison of Approaches to Large-Scale Data Analysis: MapReduce vs. DBMS Benchmarks
Overview The following information is meant to provide documentation on how others can recreate the benchmark trials used in our SIGMOD 2009 paper. Our experiments were conducted on a 100-node cluster at the University of Wisconsin-Madison; each node had a single 2.40 GHz Intel Core 2 Duo processor running 64-bit Red Hat Enterprise Linux 5 (kernel version 2.6.18) with 4GB RAM and two 250GB SATA-I
MapReduce: A Minor Step Forward – Perspectives
Disclaimer: The opinions expressed here are my own and do not necessarily represent those of current or past employers. Recent CommentsMatt on Seagate HAMRJames Hamilton on Seagate HAMRMatt on Seagate HAMRJames Hamilton on Cost of Power in Large-Scale Data CentersChris K. on Cost of Power in Large-Scale Data CentersJames Hamilton on Cost of Power in Large-Scale Data CentersRob F on Cost of Power i
第1回 分散処理を隠蔽し、大規模開発を可能に
筆者らは、オープンソースソフトウエアの分散処理ミドルウエア「Hadoop」を、基幹系のバッチ処理システムに適用するためのフレームワーク「Asakusa」を開発した。AsakusaはHadoopと同様に、オープンソースソフトウエアとして公開する。公開日は、本連載の4回目をお届けする2011年3月31日の予定である。 Asakusaを使うことでHadoopによる分散処理のメリットを享受することが可能となり、これまでRDBMSを利用していた場合と比べて、多くのケースでバッチ処理システムの性能を大幅に向上することができる。筆者らが実際に構築を支援したシステムでは、それまで4時間かかっていた処理が数分で終わるようなケースも出てきている。 盛んに報道されているように、Hadoopはすでに多くの導入実績がある。ただしその用途は、ログ分析システムやレコメンデーションエンジンなどのビジネスインテリジェンス（
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
20130201 MapReduce Design Patterns
bitbucketの使い方
Hands on MapR -- Viadea
A Study of Skew in MapReduce Applications
https://www.comp.nus.edu.sg/~epic/mrbenchmark.htm
Special Big Thinker Event: Michael J. Carey | Yahoo! Labs