Kafka with AVRO vs., Kafka with Protobuf vs., Kafka with JSON Schema
Kafka serialisation schemes — playing with AVRO, Protobuf, JSON Schema in Confluent Streaming Platform. The code for these examples available at https://github.com/saubury/kafka-serialization Apache Avro was has been the default Kafka serialisation mechanism for a long time. Confluent just updated their Kafka streaming platform with additional support for serialising data with Protocol buffers (or
Page Not Found
Sorry, but the page you were trying to view does not exist — perhaps you can try searching for it below.
Using schema auto-detection | BigQuery | Google Cloud
Send feedback Stay organized with collections Save and categorize content based on your preferences. Using schema auto-detection Schema auto-detection Schema auto-detection enables BigQuery to infer the schema for CSV, JSON, or Google Sheets data. Schema auto-detection is available when you load data into BigQuery and when you query an external data source. When auto-detection is enabled, BigQuery
More Than Just a Schema Store
This post is part of a series covering Yelp's real-time streaming data infrastructure. Our series explores in-depth how we stream MySQL and Cassandra data at real-time, how we automatically track & migrate schemas, how we process and transform streams, and finally how we connect all of this into data stores like Redshift, Salesforce, and Elasticsearch. Read the posts in the series: Billions of Mes
File Format Benchmark - Avro, JSON, ORC & Parquet
This document summarizes a benchmark study of file formats for Hadoop, including Avro, JSON, ORC, and Parquet. It found that ORC with zlib compression generally performed best for full table scans. However, Avro with Snappy compression worked better for datasets with many shared strings. The document recommends experimenting with the benchmarks, as performance can vary based on data characteristic
Capturing changes from MySQL
Change data capture is a hot topic. Debezium’s goal is to make change data capture easy for multiple DBMSes, but admittedly we’re still a young open source project and so far we’ve only released a connector for MySQL with a connector for MongoDB that’s just around the corner. So it’s great to see how others are using and implementing change data capture. In this post, we’ll review Yelp’s approach
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon 2015
Hadoop Summit June 2016 The landscape for storing your big data is quite complex, with several competing formats and different implementations of each format. Understanding your use of the data is critical for picking the format. Depending on your use case, the different formats perform very differently. Although you can use a hammer to drive a screw, it isn’t fast or easy to do so. The use cases
Ramblings of a distributed computing programmer
Understanding how Parquet integrates with Avro, Thrift and Protocol Buffers Parquet is a new columnar storage format that come out of a collaboration between Twitter and Cloudera. Parquet’s generating a lot of excitement in the community for good reason - it’s shaping up to be the next big thing for data storage in Hadoop for a number of reasons: It’s a sophisticated columnar file format, which me
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
GitHub - mozilla/jsonschema-transpiler: Compile JSON Schema into Avro and BigQuery schemas
GitHub - chenrui333/rules_avro: 🍃 bazel rules for generating code from avro schemas
Democratizing NiFi Record Processors with automatic Schemas inference
nifi/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/InferAvroSchema.java at master · apache/nifi
InferAvroSchema
Open-Sourcing Yelp's Data Pipeline
Avro Schema Registry
Avro Schema Registry with Apache Atlas for Streaming Data Management
schematizer/README.md at master · Yelp/schematizer
GitHub - spotify/gcs-tools: GCS support for avro-tools, parquet-tools and protobuf
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive
Using the Kite Command Line Interface to Create a Dataset