タグ

rcfileとlz4に関するyassのブックマーク (1)

  • Scaling the Facebook data warehouse to 300 PB

    At Facebook, we have unique storage scalability challenges when it comes to our data warehouse. Our warehouse stores upwards of 300 PB of Hive data, with an incoming daily rate of about 600 TB. In the last year, the warehouse has seen a 3x growth in the amount of data stored. Given this growth trajectory, storage efficiency is and will continue to be a focus for our warehouse infrastructure. There

    Scaling the Facebook data warehouse to 300 PB
    yass
    yass 2014/04/20
    " we evolved ORCFile to provide a significant boost in compression ratios over RCFile on our warehouse data, going from 5x to 8x. Additionally, on a large representative set of queries and data from our warehouse, we found that the Facebook ORCFile writer is 3x better on average than ORCFile. "
  • 1