Jepsen: MongoDB 3.6.4
In February 2017, we discussed data loss and fixes in MongoDB 3.4.0-rc3’s v0 and v1 replication protocols. In this Jepsen report, we will verify that MongoDB 3.6.4’s sharded clusters offer comparable safety to non-sharded deployments. We’ll also discuss MongoDB’s new support for causal consistency (CC) in version 3.6.4 and 4.0.0-rc1, and show that sessions prevent anomalies so long as user stick t
Hexing the technical interview
Previously: Reversing the technical interview. Long ago, on Svalbard, when you were a young witch of forty-three, your mother took your unscarred wrists in her hands, and spoke: Vidrun, born of the sea-wind through the spruce Vidrun, green-tinged offshoot of my bough, joy and burden of my life Vidrun, fierce and clever, may our clan’s wisdom be yours: Never read Hacker News But Hacker News has rea
Jepsen: Dgraph 1.0.2
Dgraph is a distributed graph database which uses Raft for per-shard replication and a custom transactional protocol, based on Omid, Reloaded, for snapshot-isolated cross-shard transactions. Dgraph claimed to offer snapshot isolation, per-client monotonicity, and linearizability. However, in Dgraph 1.0.2 through 1.0.6, we found multiple deadlocks & crashes in the cluster join and node recovery pro
Consistency Models
This clickable map (adapted from Bailis, Davidson, Fekete et al and Viotti & Vukolic) shows the relationships between common consistency models for concurrent systems. Arrows show the relationship between consistency models. For instance, strict serializable implies both serializability and linearizability, linearizability implies sequential consistency, and so on. Colors show how available each m
Asynchronous replication with failover
In response to my earlier post on Redis inconsistency, Antirez was kind enough to help clarify some points about Redis Sentinel’s design. First, I’d like to reiterate my respect for Redis. I’ve used Redis extensively in the past with good results. It’s delightfully fast, simple to operate, and offers some of the best documentation in the field. Redis is operationally predictable. Data structures a
Jepsen: Aerospike 3.99.0.3
Aerospike is a high-performance distributed document store. Following up on our 2015 analysis, we explored Aerospike’s new strong-consistency mode, which offers linearizable operations on single records. We confirmed two documented flaws in Aerospike’s homegrown replication algorithm. First, it can lose updates when more than k nodes crash (either concurrently or in sequence). Second, when either
Jepsen: Riak
Previously in Jepsen, we discussed MongoDB. Today, we’ll see how last-write-wins in Riak can lead to unbounded data loss. So far we’ve examined systems which aimed for the CP side of the CAP theorem, both with and without failover. We learned that primary-secondary failover is difficult to implement safely (though it can be done; see, for example, ZAB or Raft). Now I’d like to talk about a very di
Jepsen: Hazelcast 3.8.3
Hazelcast is a distributed in-memory data grid, providing shared data structures for distributed systems. We show that many of Hazelcast’s distributed data structures are unsafe in the presence of network partitions: updates to maps can be lost, unique IDs may not be unique, atomic objects are not atomic, locks aren’t exclusive, and queues can forget about enqueued elements. Stale and dirty reads
Jepsen: Aerospike
Previously, on Jepsen, we reviewed Elasticsearch’s progress in addressing data-loss bugs during network partitions. Today, we’ll see Aerospike 3.5.4, an “ACID database”, react violently to a basic partition. [Update, 2018-03-07] See the followup analysis of 3.99.0.3 Aerospike is a high-performance, distributed, schema-less, KV store, often deployed in caching, analytics, or ad tech environments. I
Tag: Jepsen
People keep asking why Jepsen is written in Clojure, so I figure it’s worth having a referencable answer. I’ve programmed in something like twenty languages. Why choose a Weird Lisp? Jepsen is built for testing concurrent systems–mostly databases. Because it tests concurrent systems, the language itself needs good support for concurrency. Clojure’s immutable, persistent data structures make it eas
Aphyr - Posts
I’m in the process of replacing a old radiator system with a centrally-ducted, air-source heat pump system with electric resistive backup heat. I’ve found that the default ecobee algorithm seems to behave surprisingly poorly for this system, and wanted to write up some of the settings that I’ve found yield better behavior. A disclaimer. I’m not an HVAC professional. I have two decades in software
Comments on "You Do it Too"
In response to You Do It Too: Forfeiting Partition Tolerance in Distributed Systems, I’d like to remind folks of a few things around CAP. Partition intolerance does not mean that partitions cannot happen, it means partitions are not supported. Specifically, partition-intolerant systems must sacrifice invariants when partitions occur. Which invariants? By Gilbert & Lynch, either the system allows n
Jepsen: Chronos
Chronos is a distributed task scheduler (cf. cron) for the Mesos cluster management system. In this edition of Jepsen, we’ll see how simple network interruptions can permanently disrupt a Chronos+Mesos cluster Chronos relies on Mesos, which has two flavors of node: master nodes, and slave nodes. Ordinarily in Jepsen we’d refer to these as “primary” and “secondary” or “leader” and “follower” to avo
Jepsen: MariaDB Galera Cluster
There’s a neat kind of symmetry here: P1 and P2 are duals of each other, preventing a read from seeing an uncommitted write, and preventing a write from clobbering an uncommitted read, respectively. P0 prevents two writes from stepping on each other, and we could imagine its dual r1(x) … r2(x)–but since reads don’t change the value of x they commute, and we don’t need to prevent them from interlea
Strong consistency models
Update, 2018-08-24: For a more complete, formal discussion of consistency models, see jepsen.io. Network partitions are going to happen. Switches, NICs, host hardware, operating systems, disks, virtualization layers, and language runtimes, not to mention program semantics themselves, all conspire to delay, drop, duplicate, or reorder our messages. In an uncertain world, we want our software to mai
Jepsen: RethinkDB 2.1.5
In this Jepsen report, we’ll verify RethinkDB’s support for linearizable operations using majority reads and writes, and explore assorted read and write anomalies when consistency levels are relaxed. This work was funded by RethinkDB, and conducted in accordance with the Jepsen ethics policy. RethinkDB is an open-source, horizontally scalable document store. Similar to MongoDB, documents are hiera
Jepsen: Percona XtraDB Cluster
Percona’s CTO Vadim Tkachenko wrote a response to my Galera Snapshot Isolation post last week. I think Tkachenko may have misunderstood some of my results, and I’d like to clear those up now. I’ve ported the MariaDB tests to Percona XtraDB Cluster, and would like to confirm that using exclusive write locks on all reads, as Tkachenko recommends, can recover serializable histories. Finally, we’ll ad
Tag: Jepsen
People keep asking why Jepsen is written in Clojure, so I figure it’s worth having a referencable answer. I’ve programmed in something like twenty languages. Why choose a Weird Lisp? Jepsen is built for testing concurrent systems–mostly databases. Because it tests concurrent systems, the language itself needs good support for concurrency. Clojure’s immutable, persistent data structures make it eas
Jepsen: RethinkDB 2.2.3 reconfiguration
In the previous Jepsen analysis of RethinkDB, we tested single-document reads, writes, and conditional writes, under network partitions and process pauses. RethinkDB did not exhibit any nonlinearizable histories in those tests. However, testing with more aggressive failure modes, on both 2.1.5 and 2.2.3, has uncovered a subtle error in Rethink’s cluster membership system. This error can lead to st
The trouble with timestamps
Some folks have asked whether Cassandra or Riak in last-write-wins mode are monotonically consistent, or whether they can guarantee read-your-writes, and so on. This is a fascinating question, and leads to all sorts of interesting properties about clocks and causality. There are two families of clocks in distributed systems. The first are often termed wall clocks, which correspond roughly to the t
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
