Apache Kafka is one of the most useful OSS for the message hub in micro services and can easily connect various data sources. This feature can be appli…
Apache Kafka for Microservices
Apache Kafka is one of the most useful OSS for the message hub in micro services and can easily connect various data sources. This feature can be appli…
Jepsen: Kafka
In the last Jepsen post, we learned about NuoDB. Now it’s time to switch gears and discuss Kafka. Up next: Cassandra. Kafka is a messaging system which provides an immutable, linearizable, sharded log of messages. Throughput and storage capacity scale linearly with nodes, and thanks to some impressive engineering tricks, Kafka can push astonishingly high volume through each node; often saturating
A Few Notes on Kafka and Jepsen
About I am the co-founder of Confluent and also the co-creator of Apache Kafka as well as various other open source projects. Kyle has a good write-up on replication and partitions in Kafka. I am a big fan of this methodology (empiricism FTW), though it is a bit uncomfortable to watch ones own project strapped to the apparatus. Kyle’s explanation is pretty clear, so I have only a few things to add
Strimzi - Apache Kafka on Kubernetes
Kafka on Kubernetes in a few minutes Strimzi provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations. Use the Quick Starts to get started now! Secure by Default Built-in security TLS, SCRAM-SHA, and OAuth authentication Automated Certificate Management Simple yet Configurable NodePort, Load balancer and Ingress options Rack awareness for HA Use dedicated no
GitHub - line/decaton: High throughput asynchronous task processing on Apache Kafka
Decaton is a streaming task processing framework built on top of Apache Kafka. It is designed to enable "concurrent processing of records consumed from one partition" which isn't possible in many Kafka consumer frameworks. Here is the list of property that Decaton enables by default: Concurrent (either multi-threaded or asynchronous) processing of records consumed from one partition Preserve order
Cooperative Rebalancing in the Kafka Consumer, Streams & ksqlDB | JP
Everyone wants their infrastructure to be highly available, and ksqlDB is no different. But crucial properties like high availability don’t come without a thoughtful, rigorous design. We thought hard about how to achieve this while maintaining at-least-once and exactly-once processing guarantees. To learn how we did it, it’s time to peel back the layers below ksqlDB and get your hands dirty with t
High Available Task Scheduling — Design using Kafka and Kafka Streams
In any Enterprise or Cloud application, Task scheduling is a key requirement. A highly available and fault tolerant task scheduling will help you to improve your business goals. A classic task scheduling infrastructure is typically backed by databases. The instances/service that performs the scheduling, loads the task definitions from the database into memory and performs the task scheduling. This
Amazon Web Services
Home Toggle submenu for: AWS Summits AWS Summit ANZ AWS Summit ASEAN AWS Summit India AWS Summit Seoul AWS Summit Tokyo Toggle submenu for: Events/Webinars AWS Innovate - AI/ML Edition AWS Innovate - Data Edition AWS Innovate - Modern Apps Edition AWSome Day Online Conference AWS Builders Online Series Toggle submenu for: Hot Topics AI and Machine Learning Migration and Modernization Data and Anal
Kafka in a Nutshell
Kafka is a messaging system. That’s it. So why all the hype? In reality messaging is a hugely important piece of infrastructure for moving data between systems. To see why, let’s look at a data pipeline without a messaging system. This system starts with Hadoop for storage and data processing. Hadoop isn’t very useful without data so the first stage in using Hadoop is getting data in. Bringing Dat
え、まって。その並列分散処理、Kafkaのしくみでもできるの？ Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
1. © 2019 NTT DATA Corporation 【テクノロジー】 え、まって。その並列分散処理、Kafkaのしくみでもできるの？ ～Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理～ NTTデータ 技術革新統括本部 システム技術本部 生産技術部 インテグレーション技術センタ データ活用チーム 佐々木 徹 2019/09/05 NTTデータ テクノロジーカンファレンス 2019 2. 2 © 2019 NTT DATA Corporation 氏名/所属 – 佐々木 徹（ささき とおる) – NTTデータ 技術革新統括本部 システム技術本部 生産技術部 インテグレーション技術センタ 業務内容 – Apache Spark, Apache Kafkaの導入や技術サポートなど オープンソースへの貢献 – 複数のオープンソースプロダクト(Hadoop
LINEの大規模データパイプラインを支える、Apache Kafkaプラットフォームの運用の裏側 - ログミーTech
LINEの大規模データパイプラインを支える、Apache Kafkaプラットフォームの運用の裏側 Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messages 2018年11月21日、LINE株式会社が主催するエンジニア向け技術カンファレンス「LINE DEVELOPER DAY 2018」が開催されました。4度目の開催となる今回のテーマは「Next LINE」。メッセージアプリだけでなく、さまざまなサービスの開発や新たな技術領域への投資を行っているLINEが目指すビジョンについて、エンジニアたちの技術的知見や挑戦を通して紹介します。セッション「Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messages
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Why run Apache Kafka on Kubernetes?
GitHub - zz85/kafka-streams-viz: Kafka Streams Topology Sketch Diagram Visualization
Debugging with kafkacat
GitHub - kawamuray/kmql: Query Kafka cluster's metadata using SQL
Apache Kafka: Docker Container and examples in Python
Kafka for Engineers
https://pages.awscloud.com/rs/112-TZM-766/images/CUS-47_AWS_Summit_Online_2020_Repro-inc.pdf
https://docs.confluent.io/current/installation/docker/installation/replicator.html
Kafka Replication: The case for MirrorMaker 2.0 - Cloudera Blog