GitHub - yahoo/TensorFlowOnSpark: TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
TensorFlowOnSpark brings scalable deep learning to Apache Hadoop and Apache Spark clusters. By combining salient features from the TensorFlow deep learning framework with Apache Spark and Apache Hadoop, TensorFlowOnSpark enables distributed deep learning on a cluster of GPU and CPU servers. It enables both distributed TensorFlow training and inferencing on Spark clusters, with a goal to minimize t
spark/KinesisWordCountASL.scala at master · apache/spark · GitHub
* Licensed to the Apache Software Foundation (ASF) under one or more
[SPARK-7076][SPARK-7077][SPARK-7080][SQL] Use managed memory for aggregations by JoshRosen · Pull Request #5725 · apache/spark
This patch adds managed-memory-based aggregation to Spark SQL / DataFrames. Instead of working with Java objects, this new aggregation path uses sun.misc.Unsafe to manipulate raw memory. This reduces the memory footprint for aggregations, resulting in fewer spills, OutOfMemoryErrors, and garbage collection pauses. As a result, this allows for higher memory utilization. It can also result in better
![[SPARK-7076][SPARK-7077][SPARK-7080][SQL] Use managed memory for aggregations by JoshRosen · Pull Request #5725 · apache/spark](https://cdn-ak-scissors.b.st-hatena.com/image/square/6e623786961cd05f3663bf734989458210997fe7/height=288;version=1;width=512/https%3A%2F%2Fopengraph.githubassets.com%2Ff9a33e016066fde03a4c0fd4c5099d7c44c8e1d9503dc74b251c703a36cce8cb%2Fapache%2Fspark%2Fpull%2F5725)
[SPARK-7078] [SPARK-7079] Binary processing sort for Spark SQL by JoshRosen · Pull Request #6444 · apache/spark
This patch adds a cache-friendly external sorter which operates on serialized bytes and uses this sorter to implement a new sort operator for Spark SQL and DataFrames. Overview of the new sorter The new sorter design is inspired by Alphasort and implements a key-prefix optimization in order to improve the cache friendliness of the sort. In naive sort implementations, the sorting algorithm operates
![[SPARK-7078] [SPARK-7079] Binary processing sort for Spark SQL by JoshRosen · Pull Request #6444 · apache/spark](https://cdn-ak-scissors.b.st-hatena.com/image/square/ae8ae3b51b8f47b21440347f84e94f7339b715d8/height=288;version=1;width=512/https%3A%2F%2Fopengraph.githubassets.com%2F38dd3170cd95cc1dea643446e7ef8e77ec33c8557dc0df8e66a10070390df1a3%2Fapache%2Fspark%2Fpull%2F6444)
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
spark/python at master · apache/spark
docker-stacks/all-spark-notebook at main · jupyter/docker-stacks
Apache Spark Notes
GitHub - intel-analytics/ipex-llm: Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, etc.) on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max); seamlessly integ
GitHub - yahoo/CaffeOnSpark: Distributed deep learning on Hadoop and Spark clusters.
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala
GitHub - databricks/spark-sql-perf
[SPARK-9020][SQL] Support mutable state in code gen expressions by cloud-fan · Pull Request #7392 · apache/spark![[SPARK-9020][SQL] Support mutable state in code gen expressions by cloud-fan · Pull Request #7392 · apache/spark](https://cdn-ak-scissors.b.st-hatena.com/image/square/9f92fb5a682d6a4a1aa950df91652242226c5f97/height=288;version=1;width=512/https%3A%2F%2Fopengraph.githubassets.com%2F98bac3f8a40c8387d7db90305ae2e7f16a4ed6c442c82d4f47d47160773c904a%2Fapache%2Fspark%2Fpull%2F7392)
GitHub - tmalaska/SparkStreaming.Sessionization: NRT Sessionization with Spark Streaming landing on HDFS and putting live stats in HBase
GitHub - tmalaska/Spark.ProdictBehaviorBasedOnPastActives: This is an example of how to do window analysis with Spark
GitHub - sryza/spark-timeseries: A library for time series analysis on Apache Spark