Hourglass: a Library for Incremental Processing on Hadoop
Slides from my talk at IEEE BigData 2013 presenting our paper "Hourglass: a Library for Incremental Processing on Hadoop" Abstract: Hadoop enables processing of large data sets through its relatively easy-to-use semantics. However, jobs are often written inefficiently for tasks that could be computed incrementally due to the burdensome incremental state management for the programmer. This paper in
DataFu 1.0
As we can see, the DataFu version provides a noticable improvement in both metrics. Glad to know that work wasn't all for naught. Creating a custom purpose UDF Many UDFs, such as those presented in the previous section, are general purpose. DataFu serves to collect these UDFs and make sure they are tested and easily available. If you are writing such a UDF, then we will happily accept contribution
Building Data Products at LinkedIn with DataFu
This document discusses tools and techniques used at LinkedIn for building data products, including Pig, Java, R, Hive, Voldemort, Kafka, and Azkaban. It describes how LinkedIn uses Pig extensively for its conciseness and expressiveness. DataFu is introduced as an open source library of user-defined functions (UDFs) for Pig that was created to share useful UDFs developed by different teams at Link
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
処理を実行中です
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
User Engagement Powered By Apache Pig and Hadoop
