* Licensed to the Apache Software Foundation (ASF) under one or more
spark/KinesisWordCountASL.scala at master · apache/spark · GitHub
* Licensed to the Apache Software Foundation (ASF) under one or more
[SPARK-7076][SPARK-7077][SPARK-7080][SQL] Use managed memory for aggregations by JoshRosen · Pull Request #5725 · apache/spark
This patch adds managed-memory-based aggregation to Spark SQL / DataFrames. Instead of working with Java objects, this new aggregation path uses sun.misc.Unsafe to manipulate raw memory. This reduces the memory footprint for aggregations, resulting in fewer spills, OutOfMemoryErrors, and garbage collection pauses. As a result, this allows for higher memory utilization. It can also result in better
![[SPARK-7076][SPARK-7077][SPARK-7080][SQL] Use managed memory for aggregations by JoshRosen · Pull Request #5725 · apache/spark](https://cdn-ak-scissors.b.st-hatena.com/image/square/6e623786961cd05f3663bf734989458210997fe7/height=288;version=1;width=512/https%3A%2F%2Fopengraph.githubassets.com%2Ff9a33e016066fde03a4c0fd4c5099d7c44c8e1d9503dc74b251c703a36cce8cb%2Fapache%2Fspark%2Fpull%2F5725)
Spark Release 1.3.0 | Apache Spark
Spark 1.3.0 is the fourth release on the 1.X line. This release brings a new DataFrame API alongside the graduation of Spark SQL from an alpha project. It also brings usability improvements in Spark’s core engine and expansion of MLlib and Spark Streaming. Spark 1.3 represents the work of 174 contributors from more than 60 institutions in more than 1000 individual patches. To download Spark 1.3 vi
MLlib: RDD-based API - Spark 3.5.1 Documentation
MLlib: RDD-based API This page documents sections of the MLlib guide for the RDD-based API (the spark.mllib package). Please see the MLlib Main Guide for the DataFrame-based API (the spark.ml package), which is now the primary API for MLlib. Data types Basic statistics summary statistics correlations stratified sampling hypothesis testing streaming significance testing random data generation Class
RDD Programming Guide - Spark 3.5.1 Documentation
RDD Programming Guide Overview Linking with Spark Initializing Spark Using the Shell Resilient Distributed Datasets (RDDs) Parallelized Collections External Datasets RDD Operations Basics Passing Functions to Spark Understanding closures Example Local vs. cluster modes Printing elements of an RDD Working with Key-Value Pairs Transformations Actions Shuffle operations Background Performance Impact
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
処理を実行中です
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala
[SPARK-9020][SQL] Support mutable state in code gen expressions by cloud-fan · Pull Request #7392 · apache/spark![[SPARK-9020][SQL] Support mutable state in code gen expressions by cloud-fan · Pull Request #7392 · apache/spark](https://cdn-ak-scissors.b.st-hatena.com/image/square/9f92fb5a682d6a4a1aa950df91652242226c5f97/height=288;version=1;width=512/https%3A%2F%2Fopengraph.githubassets.com%2F98bac3f8a40c8387d7db90305ae2e7f16a4ed6c442c82d4f47d47160773c904a%2Fapache%2Fspark%2Fpull%2F7392)
[SPARK-5006] spark.port.maxRetries doesn't work - ASF JIRA
Apache Spark™ - Unified Engine for large-scale data analytics
