Best splittable compression for Hadoop input = bz2?

We've realized a bit too late that archiving our files in GZip format for Hadoop processing isn't such a great idea. GZip isn't splittable, and for reference, here are the problems which I won't repeat: Very basic question about Hadoop and compressed input files Hadoop gzip compressed files Hadoop gzip input file using only one mapper Why can't hadoop split up a large text file and then compress t

[HHR]

gzipでの様々な問題へのリンクとbz2,LZO,LZ4について。bz2はsplittableだがCPUリソースを喰う。LZO,LZ4はCPUリソース当たりの圧縮効率が良いがindex作成、専用ツールの使用などhadoop内ルールなガラパゴス要素がある。

[wlbhiro]

]