Managing Spark Partitions with Coalesce and Repartition

Spark splits data into partitions and executes computations on the partitions in parallel. You should understand how data is partitioned and when you need to manually adjust the partitioning to keep your Spark computations running efficiently. Understanding memory partitioning is absolutely critical for Spark programmers. This post covers memory partitioning and you should check out Beautiful Spar