File Format Benchmark - Avro, JSON, ORC & Parquet

This document summarizes a benchmark study of file formats for Hadoop, including Avro, JSON, ORC, and Parquet. It found that ORC with zlib compression generally performed best for full table scans. However, Avro with Snappy compression worked better for datasets with many shared strings. The document recommends experimenting with the benchmarks, as performance can vary based on data characteristic

[rsakamot]

データによってもちろん変わるが、parquetは意外と遅いのか..。GCも結構頻発してしまう。avroも早いが、hiveで使うにもスキーマ定義が面倒。ORCを選んで置けばそれほどズレがなさそう。