Don't use Hadoop - your data isn't that big

"So, how much experience do you have with Big Data and Hadoop?" they asked me. I told them that I use Hadoop all the time, but rarely for jobs larger than a few TB. I'm basically a big data neophite - I know the concepts, I've written code, but never at scale. The next question they asked me. "Could you use Hadoop to do a simple group by and sum?" Of course I could, and I just told them I needed t

[kimutansk]

データが1TB位までならHadoopよりSQLやPythonスクリプトの方がよほど使いやすく効率的、と。その通りですねぇ。基本的にHadoopは限界は大きいですが効率は悪い

[heavenshell]

デカいデータでもないのに Hadoop つかうんじゃねーよというお話。

[tmatsuu]

「1TBぐらいならHadoopを使わなくても十分処理できる。まぁ5TB以上になったらHadoop使えばいいんじゃない？」うむ。邦訳が必要とされているっ！

[yass]

" If you have a single table containing many terabytes of data, Hadoop might be a good option for running full table scans on it. If you don’t have such a table, avoid Hadoop like the plague. / Hadoop does not have any conception of indexing. Hadoop has only full table scans. "

[yoppiblog]

データ量多くなければhadoopつかうなっていうおはなし