Haxplorer’s explorations » Blog Archive » Writing Ruby Map-Reduce programs for Hadoop
I wanted to make some crawlers that I wrote in ruby to run faster. Thought of running it using hadoop, and just started trying out using Hadoop Streaming for the purpose, and thought of sharing a short how-to on it since there might be others looking for it. Hadoop Streaming facilitates users to write the map and reduce code in any language and run it on a Hadoop grid easily. I assume that you ha
Yahoo! Launches World's Largest Hadoop Production Application (Hadoop and Distributed Computing at Yahoo!)
Yahoo! Launches World's Largest Hadoop Production Application (Hadoop and Distributed Computing at Yahoo!) Get StartedHomeAbout UsDeveloper Network BlogYDN Theater Frequently Asked QuestionsSupport CommunitiesGot Suggestions?Developer CentralColdFusion Developer Center.NET Developer CenterFlash Developer CenterJava Developer CenterJavaScript Developer CenterMobile Developer CenterPHP Developer Cen
Yahoo! Search running Apache Hadoop on Large Scale (by Jeremy Zawodny)
Over on the Yahoo! Hadoop blog, you can read about how the webmap team in Yahoo! Search is using the Apache Hadoop distributed computing framework. They're using over 10,000 CPU cores to build the map and processing a ton of data to do so. They end up using over 5 petabytes of raw disk storage, eventually outputting over 300 terabytes of compressed data that's used to power every single search. As
Hadoop Reactions & Announcement of Hadoop Summit at Yahoo! (by Jeremy Zawodny)
Hadoop Reactions & Announcement of Hadoop Summit at Yahoo! I have to say, I'm impressed at the coverage that news about our production Hadoop deployment in Yahoo! Search got yesterday (and today). Here's a quick list of the stories. Yahoo! Launches World's Largest Hadoop Production Application by Eric Baldeschwieler Yahoo! Search running Apache Hadoop on Large Scale by me Open source grid computin
Self-Service, Prorated Supercomputing Fun!
By Derek Gottfrid November 1, 2007 5:30 pm November 1, 2007 5:30 pm As part of eliminating TimesSelect, The New York Times has decided to make all the public domain articles from 1851–1922 available free of charge. These articles are all in the form of images scanned from the original paper. In fact from 1851–1980, all 11 million articles are available as images in PDF format. To generate a PDF ve
Yahoo's Doug Cutting on MapReduce and the Future of Hadoop
How to Use Multiple GitHub Accounts Git is a popular tool for version control in software development. It is not uncommon to use multiple Git accounts. Correctly configuring and switching Git accounts is challenging. In this article, we show what Git provides for account configuration, its limitations, and the solution to switch accounts automatically based on a project parent directory location.
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
