タグ

arrowとSparkに関するkimutanskのブックマーク (2)

  • Speeding up PySpark with Apache Arrow

    Published 26 Jul 2017 By BryanCutler Bryan Cutler is a software engineer at IBM’s Spark Technology Center STC Beginning with Apache Spark version 2.3, Apache Arrow will be a supported dependency and begin to offer increased performance with columnar data transfer. If you are a Spark user that prefers to work in Python and Pandas, this is a cause to be excited over! The initial work is limited to c

    Speeding up PySpark with Apache Arrow
    kimutansk
    kimutansk 2017/07/28
    spark.sql.execution.arrow.enableがSpark2.3.0からついに使用可能になりますか。正式版はまだ先ですが、ようやくお手軽に使えるようになってきますかね。
  • Cloudera Blog

    Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. In recent years, the term “data lakehouse” was coined to describe this architectural pattern of tabular analytics over data in the data lake. […] Read blog post

    kimutansk
    kimutansk 2016/02/18
    Apache ArrowでJVMプロセスと非JVMプロセス間がよりシームレスになると。ここで普通にArrow出ますか。カラムナメモリデータストアフォーマットとして様々な言語で発展する勢い?
  • 1