HDFS Erasure Codingの紹介とYahoo! JAPANにおける運用事例
ヤフー株式会社は、2023年10月1日にLINEヤフー株式会社になりました。LINEヤフー株式会社の新しいブログはこちらです。LINEヤフー Tech Blog はじめに こんにちは。Yahoo! JAPANでHadoopに携わっているエンジニアの浅沼です。Hadoopは大量のデータを分散処理するためのオープンソースソフトウエアです。この夏にリリース予定のバージョン3.0系には、HDFSの新機能であるErasure Codingが導入されます。Yahoo! JAPANではHadoopコミュニティーでErasure Codingの実装に参加してきました。本記事ではErasure Codingの仕組みを詳しく解説し、弊社での運用事例を紹介します。 既存のHDFSの課題 Hadoopの中核をなす分散ファイルシステムのHDFS（Hadoop Distributed File System）は、マス
Apache Kudu - Fast Analytics on Fast Data
Home Overview Documentation Releases Blog Community GET IN TOUCH Mailing Lists Slack Channel Events and Meetups Project Committers Ecosystem How to Contribute DEVELOPER RESOURCES GitHub Gerrit Code Review JIRA Issue Tracker SOCIAL MEDIA Twitter Reddit APACHE SOFTWARE FOUNDATION Security Sponsorship Thanks License FAQ Apache Kudu is an open source distributed data storage engine that makes fast ana
Using Flume
Read it now on the O’Reilly learning platform with a 10-day free trial. O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers. How can you get your data from frontend servers to Hadoop in near real time? With this complete reference guide, you’ll learn Flume’s rich set of features for collecting, aggregating,
Cloudera Blog
The ongoing progress in Artificial Intelligence is constantly expanding the realms of possibility, revolutionizing industries and societies on a global scale. The release of LLMs surged by 136% in 2023 compared to 2022, and this upward trend is projected to continue in 2024. Today, 44% of organizations are experimenting with generative AI, with 10% having […] Read blog post
A Guide to Checkpointing in Hadoop - Cloudera Blog
Understanding how checkpointing works in HDFS can make the difference between a healthy cluster or a failing one. Checkpointing is an essential part of maintaining and persisting filesystem metadata in HDFS. It’s crucial for efficient NameNode recovery and restart, and is an important indicator of overall cluster health. However, checkpointing can also be a source of confusion for operators of Apa
Hadoop, HBase, and Xceivers
Some of the configuration properties found in Hadoop have a direct effect on clients, such as HBase. One of those properties is called "dfs.datanode.max.xcievers", and belongs to the HDFS subproject. It defines the number of server side threads and - to some extent - sockets used for data connections. Setting this number too low can cause problems as you grow or increase utilization of your cluste
Cloudera Blog
The ongoing progress in Artificial Intelligence is constantly expanding the realms of possibility, revolutionizing industries and societies on a global scale. The release of LLMs surged by 136% in 2023 compared to 2022, and this upward trend is projected to continue in 2024. Today, 44% of organizations are experimenting with generative AI, with 10% having […] Read blog post
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
XORing Elephants: Novel Erasure Codes for Big Data ∗ Maheswaran Sathiamoorthy University of Southern California msathiam@usc.edu Megasthenis Asteris University of Southern California asteris@usc.edu Dimitris Papailiopoulos University of Texas at Austin
[HDFS-3540] Further improvement on recovery mode and edit log toleration in branch-1 - ASF JIRA
[HDFS-4350] Make enabling of stale marking on read and write paths independent - ASF JIRA
Secondary Namenode - What it really do?
[HDFS-2827] Cannot save namespace after renaming a directory above a file with an open lease - ASF JIRA
[HDFS-4425] NameNode low on available disk space - ASF JIRA
[HADOOP-8928] Add ability to reset topologies on master nodes - ASF JIRA
Balancing data equally among disks on a datanode
[HDFS-1448] Create multi-format parser for edits logs file, support binary and XML formats initially - ASF JIRA
What's New and Upcoming in HDFS - the Hadoop Distributed File System
[HDFS-3429] DataNode reads checksums even if client does not need them - ASF JIRA
[HDFS-2815] Namenode is not coming out of safemode when we perform ( NN crash + restart ) . Also FSCK report shows blocks missed. - ASF JIRA
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera