MLlib: RDD-based API This page documents sections of the MLlib guide for the RDD-based API (the spark.mllib package). Please see the MLlib Main Guide for the DataFrame-based API (the spark.ml package), which is now the primary API for MLlib. Data types Basic statistics summary statistics correlations stratified sampling hypothesis testing streaming significance testing random data generation Class

Running Spark on YARN Security Launching Spark on YARN Adding Other JARs Preparations Configuration Debugging your Application Spark Properties Available patterns for SHS custom executor log URL Resource Allocation and Configuration Overview Stage Level Scheduling Overview Important notes Kerberos YARN-specific Kerberos Configuration Troubleshooting Kerberos Configuring the External Shuffle Servic

Welcome to Py4J¶ Py4J enables Python programs running in a Python interpreter to dynamically access Java objects in a Java Virtual Machine. Methods are called as if the Java objects resided in the Python interpreter and Java collections can be accessed through standard Python collection methods. Py4J also enables Java programs to call back Python objects. Py4J is distributed under the BSD license.

Spark Programming Guide Overview Linking with Spark Initializing Spark Master URLs Deploying Code on a Cluster Resilient Distributed Datasets (RDDs) Parallelized Collections Hadoop Datasets RDD Operations Transformations Actions RDD Persistence Which Storage Level to Choose? Shared Variables Broadcast Variables Accumulators Where to Go from Here Overview At a high level, every Spark applicati