msyktのブックマーク / 2018年5月3日 - はてなブックマーク

msykt id:msykt

2018年5月3日のブックマーク (3件)

Comparison to Spark — Dask documentation
msykt 2018/05/03
DaskとSparkの比較 / “If your problems vary beyond typical ETL + SQL and you want to add flexible parallelism to existing solutions then dask may be a good fit, especially if you are already using Python and associated libraries like NumPy and Pandas.”

dask

spark
リンク
Four fails and a win at a big data stack for realtime analytics
Building a user-friendly app to analyze big data in real time (that is, keeping response times below 60 seconds) is a challenge. In the big data world, you’re either doing batch analytics where nobody really cares about query time (most businesses); or you’re doing streaming (Uber, Facebook and kin) where query time is critical, but data is only big on aggregate — each user only sees or uses a tin
msykt 2018/05/03
SparkよりもDaskが良い、という話。Pythonをメインで使ってる人から見るとこういう感じなのかな /“Obviously, I think Dask kills Spark, but the author of Dask does give a more nuanced view here if you’re interested.”

dask

spark
リンク
Data Infrastructure at In Loco
Every single tech company that operates at a very large scale will tell you about the importance of knowing how to properly handle data transport and manipulation. When providing context-aware location services for 50 million users on mobile phones all over the world, we at In Loco need to constantly re-imagine and re-invent our infrastructure such that not only our developers and overall data ana
msykt 2018/05/03
“From a technical standpoint, the presented data architecture has allowed us to scale to the throughput of over 200MBps (or over 15TB per day) produced on a daily basis on our Kafka clusters. ”

analysis

kafka

avro
リンク
- 2018年5月5日
- 2018年5月3日
- 2018年4月24日