Kafka with AVRO vs., Kafka with Protobuf vs., Kafka with JSON Schema
Kafka serialisation schemes — playing with AVRO, Protobuf, JSON Schema in Confluent Streaming Platform. The code for these examples available at https://github.com/saubury/kafka-serialization Apache Avro was has been the default Kafka serialisation mechanism for a long time. Confluent just updated their Kafka streaming platform with additional support for serialising data with Protocol buffers (or
Using schema auto-detection | BigQuery | Google Cloud
Send feedback Stay organized with collections Save and categorize content based on your preferences. Using schema auto-detection Schema auto-detection Schema auto-detection enables BigQuery to infer the schema for CSV, JSON, or Google Sheets data. Schema auto-detection is available when you load data into BigQuery and when you query an external data source. When auto-detection is enabled, BigQuery
More Than Just a Schema Store
This post is part of a series covering Yelp's real-time streaming data infrastructure. Our series explores in-depth how we stream MySQL and Cassandra data at real-time, how we automatically track & migrate schemas, how we process and transform streams, and finally how we connect all of this into data stores like Redshift, Salesforce, and Elasticsearch. Read the posts in the series: Billions of Mes
File Format Benchmark - Avro, JSON, ORC & Parquet
This document summarizes a benchmark study of file formats for Hadoop, including Avro, JSON, ORC, and Parquet. It found that ORC with zlib compression generally performed best for full table scans. However, Avro with Snappy compression worked better for datasets with many shared strings. The document recommends experimenting with the benchmarks, as performance can vary based on data characteristic
Capturing changes from MySQL
Change data capture is a hot topic. Debezium’s goal is to make change data capture easy for multiple DBMSes, but admittedly we’re still a young open source project and so far we’ve only released a connector for MySQL with a connector for MongoDB that’s just around the corner. So it’s great to see how others are using and implementing change data capture. In this post, we’ll review Yelp’s approach
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Open-Sourcing Yelp's Data Pipeline
Avro Schema Registry
Avro Schema Registry with Apache Atlas for Streaming Data Management
schematizer/README.md at master · Yelp/schematizer
DATA Data Schema and Templates
Avro - Users - Combining schemas
Flamingo
Can I split an Apache Avro schema across multiple files?
Apache Avro™ 1.7.7 Specification
GitHub - LinkedInAttic/camus: LinkedIn's previous generation Kafka to HDFS pipeline.
http://sites.computer.org/debull/A12june/pipeline.pdf