MLlib: RDD-based API This page documents sections of the MLlib guide for the RDD-based API (the spark.mllib package). Please see the MLlib Main Guide for the DataFrame-based API (the spark.ml package), which is now the primary API for MLlib. Data types Basic statistics summary statistics correlations stratified sampling hypothesis testing streaming significance testing random data generation Class

Running Spark on YARN Security Launching Spark on YARN Adding Other JARs Preparations Configuration Debugging your Application Spark Properties Available patterns for SHS custom executor log URL Resource Allocation and Configuration Overview Stage Level Scheduling Overview Important notes Kerberos YARN-specific Kerberos Configuration Troubleshooting Kerberos Configuring the External Shuffle Servic

Spark Programming Guide Overview Linking with Spark Initializing Spark Master URLs Deploying Code on a Cluster Resilient Distributed Datasets (RDDs) Parallelized Collections Hadoop Datasets RDD Operations Transformations Actions RDD Persistence Which Storage Level to Choose? Shared Variables Broadcast Variables Accumulators Where to Go from Here Overview At a high level, every Spark applicati