Introducing S3Guard: S3 Consistency for Apache Hadoop - Cloudera Blog

Synopsis This article introduces a new Apache Hadoop feature called S3Guard. S3Guard addresses one of the major challenges with running Hadoop on Amazon’s Simple Storage Service (S3), eventual consistency. We outline the problem of S3’s eventual consistency, how it affects Hadoop workloads, and explain how S3Guard works. Problem Although Apache Hadoop has support for using Amazon Simple Storage Se

[tkmoteki]

Introducing S3Guard: S3 Consistency for Apache Hadoop – Cloudera Engineering Blog

[Hash]

DynamoDB にメタデータを保存してそこで整合性を担保させることで S3 eventual consistency を乗りこなす実装の話