Top 5 mistakes when writing Spark applications

This document discusses common mistakes people make when writing Spark applications and provides recommendations to address them. It covers issues related to executor configuration, application failures due to shuffle block sizes exceeding limits, slow jobs caused by data skew, and managing the DAG to avoid excessive shuffles and stages. Recommendations include using smaller executors, increasing

[kimutansk]

Executorを最小粒度にするな／OS用リソースまで全部使い切るな／ヒープのオーバーヘッドを引け／1Executorは5コア以下にしないとHDFSあふれる、と。まだ沢山失敗事例