A Comparative Performance Evaluation of Apache Flink

Zstandard is a fast compression algorithm which you can use in Apache Spark in various way. In this talk, I briefly summarized the evolution history of Apache Spark in this area and four main use cases and the benefits and the next steps: 1) ZStandard can optimize Spark local disk IO by compressing shuffle files significantly. This is very useful in K8s environments. It’s beneficial not only when

[rjj]

むしろSparkがHadoopに比べて全然速くないのが、まあそうだろな感。Shuffleをメモリ経由でやる話は昔からあるにはあるらしいが…。