zstandard-overview.html
I recently realised that, while there is a specification for Zstandard, which describes in great details what is encoded where, there is no “overview” of the format, which would be neither too detailed nor too vague for programmers with a casual interest in data compression to understand its inner working. This blog post is an attempt to correct that. Introduction Zstandard is an LZ77-class compre
Zstd Compression For Btrfs & Squashfs Set For Linux 4.14, Already Used Within Facebook - Phoronix
Zstd Compression For Btrfs & Squashfs Set For Linux 4.14, Already Used Within Facebook Written by Michael Larabel in Linux Kernel on 8 September 2017 at 03:50 PM EDT. 13 Comments As we've been expecting, Zstd compression for Btrfs is coming with the Linux 4.14 along with Zstd support in SquashFS. Zstd/Zstandard was developed at Facebook and offers compression similar to zip/gzip but with faster sp
Brotli を用いた静的コンテンツ配信最適化と Accept-Encoding: br について | blog.jxck.io
Intro High Sierra に乗る Safari 11 で Brotli 対応がされるということで、メジャーブラウザの Brotli 対応が概ね揃うことになる。 そこで、本サイトも Brotli による静的コンテンツ配信に対応した。 brotli brotli は Google が開発した新しい圧縮形式である。 Brotli Compressed Data Format LZ77 とハフマン符号化を合わせたものであり、元々は WOFF2 の仕様の一部として作られたものが、汎用化されたものである。 過去に公開されている zopfli と比べても、さらに圧縮率が 20-26% 向上しており、解答速度は zlib 相当とされている。 この効果に寄与する特徴的な要因として、仕様に含まれる辞書が挙げられる。 Static Dictionary 圧縮アルゴリズムは、簡単に言えば頻出する一致部分
Facebook Looking To Add Zstd Support To The Linux Kernel, Btrfs - Phoronix
Facebook Looking To Add Zstd Support To The Linux Kernel, Btrfs Written by Michael Larabel in Linux Kernel on 4 August 2017 at 05:48 PM EDT. 45 Comments Zstd (also known as Zstandard) is a lossless data compression developed by Facebook that has been open-source since last year. This BSD-licensed compression algorithm aims to offer compression similar to zip/gzip but with faster speeds both for co
A guide to PNG optimization
1. Background 1.1 The PNG file format The Portable Network Graphics (PNG) is a format for storing compressed raster graphics. The compression engine is based on the Deflate method [RFC1951], designed by PKWare and originally used in PKZIP. The PNG format is defined by the PNG Specification. This specification was developed by an ad-hoc group named the PNG Development Group, and it is both an Inter
Squeezing PNG Images
Ariya Hidayat (@ariyahidayat) is an engineer/technical lead/FOSS fan/blogger/speaker. These days, his activities are mostly on software craftsmanship around HTML5, JavaScript, and other web technologies. PNG is a popular format for icons, sprites, logos, screenshots, and many others. Creating PNG images is fairly easy, but what do we need to do if we want to get the smallest file possible without
LZ4m: Taking LZ4 Compression To The Next Level - Phoronix
Happy Holidays! If you enjoy all the original Linux hardware reviews and open-source news content on Phoronix, consider joining Phoronix Premium this holiday season. For Black Friday / Cyber Monday, there is a cyber week special to go premium and enjoy an ad-free experience, native dark mode, and multi-page articles presented on a single page. LZ4m: Taking LZ4 Compression To The Next Level Written
Brotli - Wikipedia
Brotli[読み疑問点]は、Jyrki AlakuijalaとZoltán Szabadka（英語版）により開発されたオープンソースのデータ圧縮ライブラリである[3][4]。 概要[編集] BrotliはLZ77やLZ78とハフマン符号化、および2次文脈モデリング（英語版）処理の近代的な亜種がベースである。 Chrome、Opera、Firefox、Microsoft Edgeなど、Internet Explorerを除いた主要なブラウザはこのライブラリによる圧縮の恩恵を受けたWebブラウジングが可能である[5][6]。同じくGoogleにより開発されたZopfliと同じく、スイスのパンであるbrötli（英語版）に由来している[7]。 詳細[編集] Brotliの最初の発表は2015年、webフォントのオフライン圧縮を目的にしてのことであった[8]。同年9月、Googleのソフトウェア
Smaller and faster data compression with Zstandard
People are creating, sharing, and storing data at a faster rate than at any other time in history. When it comes to innovating on storing and transmitting that data, at Facebook we're making advancements not only in hardware — such as larger hard drives and faster networking equipment — but in software as well. Software helps with data processing through compression, which encodes information, lik
MyRocks: A space- and write-optimized MySQL database
At Facebook we use MySQL to manage many petabytes of data, along with the InnoDB storage engine that serves social activities such as likes, comments, and shares. While InnoDB provides great performance and reliability for a variety of workloads, it has inefficiencies on space and write amplification when used with flash storage. A few years ago, we built RocksDB, an embeddable, persistent key-val
Fighting Cancer: The Unexpected Benefit Of Open Sourcing Our Code
Fighting Cancer: The Unexpected Benefit Of Open Sourcing Our Code07/08/2015 Recently I was contacted by Dr. Igor Kozin from The Institute of Cancer Research in London. He asked about the optimal way to compile CloudFlare's open source fork of zlib. It turns out that zlib is widely used to compress the SAM/BAM files that are used for DNA sequencing. And it turns out our zlib fork is the best open s
Frame of Reference and Roaring Bitmaps
Postings listsWhile it may surprise you if you are new to search engine internals, one of the most important building blocks of a search engine is the ability to efficiently compress and quickly decode sorted lists of integers. Why is this useful? As you may know, Elasticsearch shards, which are Lucene indices under the hood, split the data that they store into segments which are regularly merged
lzturbo
LzTurbo is a high performance lz77 based compressor featuring the fastest compression/decompression. Serveral times faster than the fastest compressor (see LzTurbo benchmarks) Download: LzTurbo v1.2 for windows and linux - Parallel and linear scalable compression/decompression on multiprocessor/multicore systems. - Overlapping i/o with compression/decompression. - Parallel compression/decompressio
GZ-Sort: A utility for sorting really big files.
Nice app, but unfortunately limited to gzipped data. Would be a lo... sources are at github git clone https://github.com/keenerd/gz-sort; cd gz-sort; make; ./gz-sort -h Gz-sort sorts gzipped data files. Really really big gzipped data files that I couldn't figure out how to wrangle with gnu-sort. After heavy tweaking gnu-sort can do some very large files indeed, but with poor big-O disk patterns. I
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
GitHub - chadnickbok/redis-zstd-module: Magically make your Redis data smaller with Zstandard compression.
Testing PGO with LZ4 and Zstd codecs
lzturbo - Compression Benchmark
GitHub - inikep/lizard: Lizard (formerly LZ5) is an efficient compressor with very fast decompression. It achieves compression ratio that is comparable to zip/zlib and zstd/brotli (at low and medium compression levels) at decompression speed of 1000 MB/s
GitHub - jasonmoo/smlr: Re-encode jpeg images with no perceivable quality loss.
GitHub - airlift/aircompressor: A port of Snappy, LZO, LZ4, and Zstandard to Java