Hive/JoinOptimization - Hadoop Wiki

1. Map Join Optimization 1.1 Using Distributed Cache to Propagate Hashtable File Previously, when 2 large data tables need to do a join, there will be 2 different Mappers to sort these tables based on the join key and emit an intermediate file, and the Reducer will take the intermediate file as input file and do the real join work. This join mechanism (referred to as Common Join) is perfect with t

[shiumachi]

HiveのJOINがメチャ速くなった話。DistCacheの採用やJDBMの使用可否の選択によりMAPJOINは最大28倍速くなり、自動的にMAPJOINを実行したりバックアップタスクを流すことで通常JOINも最大2.6倍高速化。詳細は図を参照