Introducing the Confluent Operator: Apache Kafka® on Kubernetes
Introducing the Confluent Operator: Apache Kafka on Kubernetes Made Simple At Confluent, our mission is to put a Streaming Platform at the heart of every digital company in the world. This means, making it easy to deploy and use Apache Kafka and Confluent Platform—the de-facto Streaming Platform—across a variety of infrastructure environments. In the last few years, the rise of Kubernetes as the c
Multitenancy: Kafka clusters for everyone at LINE
Yuto Kawamura from LINE Corporation presented on their use of Apache Kafka clusters to provide multitenancy for different internal teams. They face challenges in ensuring isolation between client workloads and preventing abusive clients. Their solutions include request quotas to limit client resource usage, slow logs to identify slow requests, and changes to the broker code to pre-warm caches and
Kafka on Kubernetes - using etcd
The content of this page hasn't been updated for years and might refer to discontinued products and projects. At Banzai Cloud we are building a cloud agnostic, open source next generation CloudFoundry/Heroku-like PaaS, Pipeline, while running several big data workloads natively on Kubernetes. Apache Kafka is one of the cloud native workloads we support out-of-the-box, alongside Apache Spark and Ap
Squeezing the firehose: getting the most from Kafka compression
Gzip sounded too expensive from the beginning (especially in Go), but Snappy should have been able to keep up. We profiled our producer and it was spending just 2.4% of CPU time in Snappy compression, never saturating a single core: Even gzip was spending just 20%, and it also did not saturate CPU: It seemed that Kafka was doing something that was slowing us down. In theory, it should only decompr
Should You Put Several Event Types in the Same Kafka Topic? | Confluent
Should You Put Several Event Types in the Same Kafka Topic? If you adopt a streaming platform such as Apache Kafka, one of the most important questions to answer is: what topics are you going to use? In particular, if you have a bunch of different events that you want to publish to Kafka as messages, do you put them in the same topic, or do you split them across different topics? The most importan
Blog Posts | August 6, 2024 - November 10, 2023 | @lightbend
Webinar: Akka 24.05 release highlights Tyler Jewell, CEO, Jonas Boner, founder and CTO, and Michael Nash, CISO, delve into the value and power of these new features and enhancements. Q&A to follow. Lightbend aims to democratize distributed systems for developers Tyler Jewell, CEO of Lightbend, talks about some of the key challenges developers are up against with distributed systems and how Lightbe
Apache Kafkaを使ったアプリ設計で反省している件を正直ベースで話す
Apache Kafka: Producer, Broker and Consumer2017年は生まれて始めてApache Kafkaを本格的に業務利用（PoCではなく本番運用）した年でした。Apache Kafka的なメッセージングミドルウェアそのもののは、社内的な事情でよく使っていたのでその使い勝手に対して困惑はほとんど無かったですし、ミドルウェアとして非常に安定しているため、Kafkaクラスタそのものでの不具合らしい不具合が発生したことは一度もありませんでした。 しかし、Kafkaのトピック設計などに関してのベストプラクティスは事例ベースでもあまり見かけたことがなく、チームメンバーと悩むことも多かったです。このストーリーでは、主にKafkaを利用したアプリ設計で考えたことや失敗したことを振り返りつつ共有します。なお、パーティション数や各種バッファサイズなどのチューニング要素は今回取
Improving Kafka at-least-once performance at Uber
At Uber, we are seeing an increasing demand for Kafka at-least-once delivery (asks=all). So far, we are running a dedicated at-least-once Kafka cluster with special settings. With a very low workload, the dedicated at-least-once cluster has been working well for more than a year. When trying to allow at-least-once producing on the regular Kafka clusters, the producing performance was the main conc
Kafka at half the price with JBOD setup
The talk introduces JBOD setup for Apache Kafka and shows how LinkedIn can save more than 30% storage cost in Kafka by adopting JBOD setup. The talk is given during the LinkedIn Streaming meetup in May, 2017.Read less
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications 1) Apache Kafka is a distributed system with many moving parts to monitor, including brokers, topics, partitions, and the applications that use Kafka. It is critical to monitor Kafka performance to ensure high availability and catch problems early. 2) Key metrics to monitor include partition replication, broker resource usa
Apache Spark Streaming + Kafka 0.10 with Joan Viladrosariera
Spark Streaming has supported Kafka since it’s inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, wi
Apache Kafka Goes 1.0
It has been seven years since we first set out to create the distributed streaming platform we know now as Apache Kafka®. Born initially as a highly scalable messaging system, Apache Kafka has evolved over the years into a full-fledged distributed streaming platform for publishing and subscribing, storing, and processing streaming data at scale and in real-time. Since we first open-sourced Apache
Kafkaコンシューマのリバランスはいつ行われるか - n-agetsumaの日記
Kafkaのコンシューマグループに、コンシューマを追加すると、各コンシューマがどのパーティションを読み取るか再割り当てするリバランスが行われる。 Kafka0.11.0.1を対象に、以下の観点でKafkaの実装を調べてみた結果をまとめる。 リバランスはいつ行われるのか なぜ新規コンシューマ追加しても、すぐにメッセージ配信が始まらないのか リバランスが起こった時に重複メッセージ処理は発生しないか リバランスはいつ行われるのか KafkaConsumer.poll()メソッドの内部で行われる。 Kafkaのコンシューマの実装はだいたい以下のようになる。コード全体はgist参照。 Kafka Comsumer Sample · GitHub KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties()); co
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Instaclustr Kafka Meetup Sydney Presentation
Kafka vs Pulsar @KafkaMeetup_20180316
Monitoring Apache Kafka
End-to-End Exactly-Once Processing in Apache Flink with Apache Kafka
KIP-101 - Alter Replication Protocol to use Leader Epoch rather than High Watermark for Truncation - Apache Kafka - Apache Software Foundation
Kafka 2018 - Securing Kafka the Right Way
https://jobs.zalando.com/tech/blog/rock-solid-kafka/index.html