Understanding HDFS Recovery Processes (Part 1) - Cloudera Blog
Having a good grasp of HDFS recovery processes is important when running or moving toward production-ready Apache Hadoop. An important design requirement of HDFS is to ensure continuous and correct operations to support production deployments. One particularly complex area is ensuring correctness of writes to HDFS in the presence of network and node failures, where the lease recovery, block recove
HDFS DataNode Scanners and Disk Checker Explained - Cloudera Blog
As many of us know, data in HDFS is stored in DataNodes, and HDFS can tolerate DataNode failures by replicating the same data to multiple DataNodes. But exactly what happens if some DataNodes’ disks are failing? This blog post explains how some of the background work is done on the DataNodes to help HDFS to manage its data across multiple DataNodes for fault tolerance. Particularly, we will explai
Improving Query Performance Using Partitioning in Apache Hive - Cloudera Blog
Improving Query Performance Using Partitioning in Apache Hive Our thanks to Rakesh Rao of Quaero, for allowing us to re-publish the post below about Quaero’s experiences using partitioning in Apache Hive. In this post, we will talk about how we can use the partitioning features available in Hive to improve performance of Hive queries. Partitions Hive is a good tool for performing queries on large
How-to: Tune Your Apache Spark Jobs (Part 1) - Cloudera Blog
Editor’s Note, January 2021: This blog post remains for historical interest only. It covers Spark 1.3, a version that has become obsolete since the article was published in 2015. For a modern take on the subject, be sure to read our recent post on Apache Spark 3.0 performance. You can also gain practical, hands-on experience by signing up for Cloudera’s Apache Spark Application Performance Tuning
How-to: Tune Your Apache Spark Jobs (Part 2) - Cloudera Blog
How resource tuning, parallelism, and data representation affect Spark 1.3 job performance. Editor’s Note, January 2021: This blog post remains for historical interest only. It covers Spark 1.3, a version that has become obsolete since the article was published in 2015. For a modern take on the subject, be sure to read our recent post on Apache Spark 3.0 performance. You can also gain practical, h
Cloudera Blog
The ongoing progress in Artificial Intelligence is constantly expanding the realms of possibility, revolutionizing industries and societies on a global scale. The release of LLMs surged by 136% in 2023 compared to 2022, and this upward trend is projected to continue in 2024. Today, 44% of organizations are experimenting with generative AI, with 10% having […] Read blog post
Cloudera Data Science Workbench: Self-Service Data Science for the Enterprise - Cloudera Blog
Cloudera Data Science Workbench: Self-Service Data Science for the Enterprise We are entering the golden age of machine learning, and it’s all about the data. As the quantity of data grows and the costs of compute and storage continue to drop, the opportunity to solve the world’s biggest problems has never been greater. Our customers already use advanced machine learning to build self-driving cars
Untangling Apache Hadoop YARN, Part 4: Fair Scheduler Queue Basics - Cloudera Blog
Untangling Apache Hadoop YARN, Part 4: Fair Scheduler Queue Basics In this installment, we provide insight into how the Fair Scheduler works, and why it works the way it does. In Part 3 of this series, you got a quick introduction to Fair Scheduler, one of the scheduler choices in Apache Hadoop YARN (and the one recommended by Cloudera). In Part 4, we will cover most of the queue properties, some
Better SLAs via Resource-preemption in YARN's CapacityScheduler - Cloudera Blog
Better SLAs via Resource-preemption in YARN’s CapacityScheduler Mayank Bansal, of EBay, is a guest contributing author of this collaborative blog. This is the 4th post in a series that explores the theme of enabling diverse workloads in YARN. See the introductory post to understand the context around all the new features for diverse workloads as part of Apache Hadoop YARN in HDP. Background In Had
Cloudera Blog
Enterprises see embracing AI as a strategic imperative that will enable them to stay relevant in increasingly competitive markets. However, it remains difficult to quickly build these capabilities given the challenges with finding readily available talent and resources to get started rapidly on the AI journey. Cloudera recently signed a strategic collaboration agreement with Amazon […] Read blog p
Flafka: Apache Flume Meets Apache Kafka for Event Processing - Cloudera Blog
The new integration between Flume and Kafka offers sub-second-latency event processing without the need for dedicated infrastructure. In this previous post you learned some Apache Kafka basics and explored a scenario for using Kafka in an online application. This post takes you a step further and highlights the integration of Kafka with Apache Hadoop, demonstrating both a basic ingestion capabilit
Cloudera Blog
The ongoing progress in Artificial Intelligence is constantly expanding the realms of possibility, revolutionizing industries and societies on a global scale. The release of LLMs surged by 136% in 2023 compared to 2022, and this upward trend is projected to continue in 2024. Today, 44% of organizations are experimenting with generative AI, with 10% having […] Read blog post
A Guide to Checkpointing in Hadoop - Cloudera Blog
Understanding how checkpointing works in HDFS can make the difference between a healthy cluster or a failing one. Checkpointing is an essential part of maintaining and persisting filesystem metadata in HDFS. It’s crucial for efficient NameNode recovery and restart, and is an important indicator of overall cluster health. However, checkpointing can also be a source of confusion for operators of Apa
How HiveServer2 Brings Security and Concurrency to Apache Hive - Cloudera Blog
How HiveServer2 Brings Security and Concurrency to Apache Hive Apache Hive was one of the first projects to bring higher-level languages to Apache Hadoop. Specifically, Hive enables the legions of trained SQL users to use industry-standard SQL to process their Hadoop data. However, as you probably have gathered from all the recent community activity in the SQL-over-Hadoop area, Hive has a few limi
Cloudera Blog
The ongoing progress in Artificial Intelligence is constantly expanding the realms of possibility, revolutionizing industries and societies on a global scale. The release of LLMs surged by 136% in 2023 compared to 2022, and this upward trend is projected to continue in 2024. Today, 44% of organizations are experimenting with generative AI, with 10% having […] Read blog post
Cloudera Blog
The ongoing progress in Artificial Intelligence is constantly expanding the realms of possibility, revolutionizing industries and societies on a global scale. The release of LLMs surged by 136% in 2023 compared to 2022, and this upward trend is projected to continue in 2024. Today, 44% of organizations are experimenting with generative AI, with 10% having […] Read blog post
Cloudera Blog
In an era where artificial intelligence (AI) is reshaping enterprises across the globe—be it in healthcare, finance, or manufacturing—it’s hard to overstate the transformation that AI has had on businesses, regardless of industry or size. At Cloudera, we recognize the urgent need for bold steps to harness this potential and dramatically accelerate the time to […] Read blog post
Managing CPU Resources in your Hadoop YARN Clusters - Cloudera Blog
This is the fourth post in a series that explores the theme of enabling diverse workloads in YARN. See the introductory post to understand the context around all the new features for diverse workloads as part of YARN in HDP 2.2. Introduction When it comes to managing resources in YARN, there are two aspects that we, the YARN platform developers, are primarily concerned with: Resource allocation: A
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Cloudera Blog
Cloudera Blog