DDSketch: A fast and fully-mergeable quantile sketch with relative-error guarantees
Summary statistics such as the mean and variance are easily maintained for large, distributed data streams, but order statistics (i.e., sample quantiles) can only be approximately summarized. There is extensive literature on maintaining quantile sketches where the emphasis has been on bounding the rank error of the sketch while using little memory. Unfortunately, rank error guarantees do not precl
GitHub - google/differential-privacy: Google's differential privacy libraries.
Note If you are unfamiliar with differential privacy (DP), you might want to go through "A friendly, non-technical introduction to differential privacy". This repository contains libraries to generate ε- and (ε, δ)-differentially private statistics over datasets. It contains the following tools. Privacy on Beam is an end-to-end differential privacy framework built on top of Apache Beam. It is inte
Algorithms for calculating variance - Wikipedia
Algorithms for calculating variance play a major role in computational statistics. A key difficulty in the design of good algorithms for this problem is that formulas for the variance may involve sums of squares, which can lead to numerical instability as well as to arithmetic overflow when dealing with large values. A formula for calculating the variance of an entire population of size N is: Usin
A better algorithm for backups and rolling logs - IFHO
In your /var/log/ you will most probably have logs that have grown too large and rolled over. Per default your system logger gzips and stores a few of the older ones and finally when they get too numerous, it just deletes them. Same thing for log handlers in most languages, for example Python’s RotatingFileHandler. Backups are usually also handled the same way, when you don’t want to store every b
Q-Digest | paper cruncher
Continuing on the same note as the Greenwald-Khanna post, we discuss another novel data structure for order statistics called Q-Digest. Compared to GK, Q-Digest is much simpler, easier to implement and extends to a distributed solution very easily. The data structure was designed with sensor networks in mind, where minimizing radio transmission overhead is of paramount importance. It originally a
Arithmetic Encoding Using Fixed-Point Math
My previous post acted as an introduction to arithmetic coding, using the 1MB Sorting Problem as a case study. I ended that post with the question of how to work with fractional values having millions of significant binary digits. The answer is that we don’t have to. Let’s work with 32-bit fixed-point math instead, and see how far that gets us. A 32-bit fixed-point number can represent fractions i
What is Multi-Armed Bandit(MAB) Testing? | VWO
Most readers of this blog would be familiar with A/B Testing. Just as a quick reminder, A/B Testing is an experiment where a random visitor to your digital property is shown a different version than the original (also called ‘control’) in an optimizer’s quest to discover an optimal version that maximizes conversions. For example, maybe it’s the red button that maximized clicks or maybe it’s the bl
Multi-armed bandit - Wikipedia
A row of slot machines in Las VegasIn probability theory and machine learning, the multi-armed bandit problem (sometimes called the K-[1] or N-armed bandit problem[2]) is a problem in which a decision maker iteratively selects one of multiple fixed choices (i.e., arms or actions) when the properties of each choice are only partially known at the time of allocation, and may become better understood
20 lines of code that will beat A/B testing every time
I know how to make and sell software online, and I can share my tips with you. Email | Twitter | LinkedIn | Comics | All articles A/B testing is used far too often, for something that performs so badly. It is defective by design: Segment users into two groups. Show the A group the old, tried and true stuff. Show the B group the new whiz-bang design with the bigger buttons and slightly different co
正規分布と正規乱数 - chmod 777 myknowledge
ノイズを付加する場合に確率密度関数として正規分布を用いることがある。 これを実際にやってみたくなったので実現方法を調べてみた。自分への覚え書きとして以下に示す。 正規分布の説明は以下を参照。 http://ja.wikipedia.org/wiki/%E6%AD%A3%E8%A6%8F%E5%88%86%E5%B8%83 正規分布を表す式は以下。 正規分布に従うノイズを発生させるためには、上式における x をノイズ量として x 軸上の各点におけるノイズを f(x) で表される確率で発生させればよい。 一方、多くのプログラミング言語で提供されているのは一様乱数(ある有限の区間を区切って、その区間内で全ての実数が同じ確率（濃度）で現れるような乱数 by Wikipedia)を発生させるメソッドである。C言語でいえば stdlib.h で提供される rand() が相当する。 0より大きく(0は
Writing Unit Tests for Sampling | Graham's Research Blog
One popular method for estimating statistical models is sampling, and in particular Gibbs sampling. As I’ve written before, sampling is based on randomness, which makes it somewhat difficult to debug. On the other hand, one popular way to ensure the quality of code is through arrays of unit tests, which are designed to test your code at a very fine-grained level (e.g. one test for each function in
How Not To Sort By Average Rating
By Evan Miller February 6, 2009 (Changes) Translations: Dutch Estonian German Russian Ukrainian PROBLEM: You are a web programmer. You have users. Your users rate stuff on your site. You want to put the highest-rated stuff at the top and lowest-rated at the bottom. You need some sort of “score” to sort by. WRONG SOLUTION #1: Score = (Positive ratings) − (Negative ratings) Why it is wrong: Supp
SZaru: Porting of excellent Sawzall aggregators.
Overview Example Install Document Source License Overview SZaru is a library to use Google's Sawzall aggregators in pure C++, Ruby and Python. Sawzall aggregators use memory efficient and one-pass algorithms to approximately compute popular statistics. For example, a simple algorithm of 'top N' computation requires O(K) memories where K means the number of unique elements. But SZaru requires only
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
GitHub - google/rappor: RAPPOR: Privacy-Preserving Reporting Algorithms
GitHub - johnmyleswhite/BanditsBook: Code for my book on Multi-Armed Bandit Algorithms
imHo