CSE 138: Distributed Systems
From the beginning of this year, I started to take lecture courses of undergrad distributed systems course at UC Santa Cruz (CSE 138) by Lindsey Kuper. It consists of 23 lectures (you can see the schedule of topics from here) and recently I’ve finished all of them. I’m not a student at UCSC but due to the COVID-19 situation, all these lectures were delivered online and are available on YouTube. So
In Search of an Understandable Consensus Algorithm (Extended Version) Diego Ongaro and John Ousterhout Stanford University Abstract Raft is a consensus algorithm for managing a replicated log. It produces a result equivalent to (multi-)Paxos, and it is as efficient as Paxos, but its structure is different from Paxos; this makes Raft more understandable than Paxos and also provides a better foundat
週刊Railsウォッチ（20210126後編）Google Cloud FunctionsがRubyをサポート、Ruby 3のパターンマッチングでポーカーゲームほか｜TechRacho by BPS株式会社
2021.01.26 週刊Railsウォッチ（20210126後編）Google Cloud FunctionsがRubyをサポート、Ruby 3のパターンマッチングでポーカーゲームほか こんにちは、hachi8833です。 各記事冒頭には⚓でパーマリンクを置いてあります: 社内やTwitterでの議論などにどうぞ 「つっつきボイス」はRailsウォッチ公開前ドラフトを（鍋のように）社内有志でつっついたときの会話の再構成です👄 お気づきの点がありましたら@hachi8833までメンションをいただければ確認・対応いたします🙇 TechRachoではRubyやRailsの最新情報などの記事を平日に公開しています。TechRacho記事をいち早くお読みになりたい方はTwitterにて@techrachoのフォローをお願いします。また、タグやカテゴリごとにRSSフィードを購読することもできます
FoundationDB: A Distributed Key-Value Store – Communications of the ACM
FoundationDB is an open-source transactional key-value store created more than 10 years ago. It is one of the first systems to combine the flexibility and scalability of NoSQL architectures with the power of ACID transactions. FoundationDB adopts an unbundled architecture that decouples an in-memory transaction management system, a distributed storage system, and a built-in distributed configurati
The Twelve-factor app is a methodology for building software-as-a-service apps that was first formulated by developers associated with Heroku. It's been ten years since the first presentation of this methodology. Despite the criticism that it is only applicable to Heroku and similar webapp services, it remains a relevant yard stick for software-as-a-service development. Some of its tenets have bee
This document contains various resources and quick definition of a lot of background information behind distributed systems. It is not complete, even though it is kinda sorta detailed. I had written it some time in 2019 when coworkers at the time had asked for a list of references, and I put together what I thought was a decent overview of the basics of distributed systems literature and concepts.
Reverse Proxy, HTTP Keep-Alive Timeout, and sporadic HTTP 502sHow HTTP Keep-Alive can cause TCP race condition These mysterious HTTP 502s happened to me already twice over the past few years. Since the amount of service-to-service communications every year goes only up, I expect more and more people to experience the same issue. So, sharing it here. TL;DR: HTTP Keep-Alive between a reverse proxy a
Database Fundamentals
About a year ago, I tried thinking which database I should choose for my next project, and came to the realization that I don't really know the differences of databases enough. I went to different database websites and saw mostly marketing and words I don't understand. This is when I decided to read the excellent books Database Internals by Alex Petrov and Designing Data-Intensive Applications by
Hydro is a high-level distributed programming framework for Rust. Hydro can help you quickly write scalable distributed services that are correct by construction. Much like Rust helps with memory safety, Hydro helps with distributed safety. Hydro also makes it easy to get started by running your distributed programs in either testing or deployment modes. Hydro is a distributed dataflow language, p
CeleryのMessage Priorities機能を利用した処理遅延の低減 - Nealle Developer's Blog
こんにちは、SREチームの宮後(@miya10kei)です。 バイクに乗っていて気持ちが良い季節になってきましたね🌸 メッセージキューを利用した非同期タスクを扱っていて、誰しも優先度順にタスクを処理したいなと思ったことがあるのではないでしょうか？ 今回はCeleryの機能を利用して実現することができたので紹介したいと思います。 Celeryってなに？ Celeryは分散メッセージキュー機能を提供するPythonベースのOSSです。 メッセージキューのBrokerとしてRedisやRabbitMQ、Amazon SQSなどを使用でき、分散環境での非同期タスクの実行を実現しています。 公式サイトを引用すると次の説明になりますね。 Celery is a simple, flexible, and reliable distributed system to process vast amou
GitHub - facundoolano/software-papers: 📚 A curated list of papers for Software Engineers
A curated list of papers that may be of interest to Software Engineering students or professionals. See the sources and selection criteria below. List of papers by topic Von Neumann's First Computer Program. Knuth (1970). Computer History; Early Programming The Education of a Computer. Hopper (1952). Recursive Programming. Dijkstra (1960). Programming Considered as a Human Activity. Dijkstra (1965
Containerize and deploy a gRPC application on AWS Fargate | Amazon Web Services
AWS Open Source Blog Containerize and deploy a gRPC application on AWS Fargate These days, building an application distributed among processes, hosts, or even networks is much easier. This is partially due to an evolution in the protocols used to negotiate between different components of an application or service. This evolution is largely the result of the popularity of both the TCP and HTTP prot
Go support for AWS X-Ray now available in AWS Distro for OpenTelemetry | Amazon Web Services
AWS Open Source Blog Go support for AWS X-Ray now available in AWS Distro for OpenTelemetry In this blog post, AWS interns Wilbert Guo and Kelvin Lo share their experience in enhancing the OpenTelemetry Go SDK to support sending traces to AWS X-Ray. These enhancements are also available in the AWS Distro for OpenTelemetry. AWS X-Ray is a service that collects data and provides tools that allow us
AWS Open Source Blog Innovating with Rust The Rust programming language is an open source project started by Mozilla Research more than a decade ago. Since then, more than 5,000 people have contributed to the Rust project, and the language has had a resounding impact on technology. Technology companies, large and small, are using and benefiting from Rust. The Rust language builds on the superpower
Valley of the Temples, Agrigento, AG, Italy. Photo by Dario Crisafulli on Unsplash.In the past few years, much has been talked and written about the “three pillars of observability”: metrics, logs, and traces. A Google search for the phrase brings up over 7,000 results, with almost every observability vendor having a blog post or an e-book on the topic. Recently, the term MELT started showing up t
？？？：雰囲気で分散システム使ってるやついる!? ？？？：いねぇよな！！！！！ むっそ：んなわけねぇよ、いるんだよ！！ここによぉ！！ (終) はじめに なんか分散システムっぽいのは使えてるけど、いまいち詳しいことは知らないってひとが多いと思います。私もです。 別に分散システムの仕組みを知らなくても問題なかったりするものですが、いざシステム障害などが起きると これそもそもの仕組み知らんと無力じゃね？ ってなったりします。(無情にもやばいバグが時々起きたりします) 今回は分散システムにわか勢による分散システムを知るのに良さそうなリンクをまとめたり感想を書いていこうかと思います。 分散システムを考え始めたきっかけ DynamoDB嘘松事件 事件前の実装 現在開発しているプロダクトで下記のような処理がありました。 (先人の過去の遺物が残っていてあまり良くないアーキテクチャになってます) フロントエ
Kubernetes README
Kubernetes READMEPlease submit for books, tutorials, or other assets that would be useful to folks using Kubernetes. PHIPPY AND FRIENDSBy: Matt Butcher, Karen Chu & Bailey Beougher The characters Phippy, Captain Kube, Goldie, and Zee and the two books are owned by The Linux Foundation, on behalf of the Cloud Native Computing Foundation, and licensed under the Creative Commons Attribution License (
February 19, 2021 This article was contributed by Paolo Bonzini Lockless algorithms are of interest for the Linux kernel when traditional locking primitives either cannot be used or are not performant enough. For this reason they come up every now and then on LWN; one of the last mentions, which prompted me to write this article series, was last July. Topics that arise even more frequently are rea
Supercharge the One Person Framework with SQLite | Fractaled Mind
NOTE: This is an edited transcript of a talk I gave at Rails World 2024. You can watch the full talk on YouTube. From its beginning, Rails has been famous for being a kind of a rocket engine that could propel your idea to astronomic heights at supersonic speed. But, at least for me, it has lately felt like I needed to be a rocket scientist to then deploy and run my full-featured application. And t
Best practices to optimize data access performance from Amazon EMR and AWS Glue to Amazon S3 | Amazon Web Services
AWS Big Data Blog Best practices to optimize data access performance from Amazon EMR and AWS Glue to Amazon S3 June 2024: This post was reviewed for accuracy and updated to cover Apache Iceberg. June 2023: This post was reviewed and updated for accuracy. Customers are increasingly building data lakes to store data at massive scale in the cloud. It’s common to use distributed computing engines, clo
.................................. E-mail: masashi.une@boj.or.jp / /2019.7 127 1. crypto-asset 1 2 3 IP Genkin, Papadopoulos, and Papamanthou [2018] Henry, Herzberg, and Kate [2018] Khalilov and Levi [2018] IP .................................. 1 virtual currency crypto-currency Financial Stability Board Financial Stability Board [2018] 2 2018 3 Tziakouris [2018] 128 /2019.7 2 3 4 2. 1 unlinkabili
No more DSLs: Implement and deploy a distributed system with a single program
No more DSLs: Implement and deploy a distributed system with a single program If you want to write a distributed system, then instead of writing a thousand different programs and configuration files in many different DSLs, you can use an approach I call "single-program systems", and write just one program. In this article, I describe how we do this at $DAYJOB, and I show some example programs. The
Yahoo! JapanThere’s clearly a LOT of activity in the Bay Area, California when it comes to vector databases! Also, there’s a large amount of variation in funding and valuations, and it’s clear that there exists no correlation between the capabilities of a database and the amount it’s funded for. Choice of programming language#Fast, responsive and scalable databases are typically written these days
2020年、世界中で猛威を振るったマルウェアといえば「Emotet」だろう。なりすましメールを介して急激に拡大し、海外はもちろん国内でも複数の企業が被害に遭い、対応に追われた。 21年1月には欧州刑事警察機構（EUROPOL）がEmotetの攻撃用サーバを制圧したと発表。攻撃は一段落した。しかし、今後も類似の攻撃やマルウェアには引き続き注意が必要だ。なぜなら、Emotetの拡大を通して攻撃者が「マルウェアフリー」という手法の効果を実感してしまったからだ。 Emotetで採用されたマルウェアフリーという手法が恐ろしいのは、従来のアンチウイルス製品では検知できないような仕組みを持っている点にある。 情報セキュリティ事業を手掛けるBlue Planet-worksの鴫原祐輔氏は「アンチウイルス製品が安全だと認識している機能やツール、検査対象に含まない形式のファイルを悪用する手口が増えています」と
Distributed cloud systems made easy Acton is a fault tolerant distributed programming platform for building mission critical systems. The easy way. Explore More ⇩ Distributed Computing built in Write programs that seamlessly run as a distributed system over an entire data center or region. All without a single line of RPC code. Durable State Acton automatically persists the state of your applicati
An In-Depth Analysis of Distributed Transaction Solutions
This article summarizes five distributed transaction solutions and analyzes their features. By Daosu This article summarizes five distributed transaction solutions and analyzes their features: XA Specification, TCC, Saga, Local-Message-Based Distributed Transactions, and Transactional-Message-Based Distributed Transactions. Considering its length, I recommend that you add it to your favorites and
5 Things I Wish I’d Known Before Starting a Design System at Spotify
Sure, this is the Spotify Design blog, and I dive a little into engineering in the article below — but design systems are cross-discipline at their heart and require a little bit from everyone and all disciplines to truly succeed! Author’s note: You’ll notice that throughout this article I use the word “customers” when talking about a design system's primary users -- your teammates. In most cases,
Amazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2 | Amazon Web Services
AWS Open Source Blog Amazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2 Large-scale, distributed compute framework migrations are not for the faint of heart. There are backwards-compatibility constraints to maintain, performance expectations to meet, scalability limits to overcome, and the omnipresent risk of introducing breaking changes to production. This all becomes especia
週刊Railsウォッチ: The Rails Foundation発足、Ruby 3.2.0 Preview 3リリース、Ruby演算子クイズほか（20221122）｜TechRacho by BPS株式会社
週刊Railsウォッチについて 各記事冒頭には🔗でパーマリンクを置いてあります: 社内やTwitterでの議論などにどうぞ 「つっつきボイス」はRailsウォッチ公開前ドラフトを（鍋のように）社内有志でつっついたときの会話の再構成です👄 お気づきの点がありましたら@hachi8833までメンションをいただければ確認・対応いたします🙏 TechRachoではRubyやRailsなどの最新情報記事を平日に公開しています。TechRacho記事をいち早くお読みになりたい方はTwitterにて@techrachoのフォローをお願いします。また、タグやカテゴリごとにRSSフィードを購読することもできます（例:週刊Railsウォッチタグ） 🔗Rails: 先週の改修（Rails公式ニュースより） 公式更新情報: Ruby on Rails — This Week in Rails: A bug
Playing the Open Source Game
You've surely read plenty about how simple is good, but what's wrong with easy? The Zig project has a non-profit foundation that needs to be managed, a community to shape, and the actual language to develop. These are all big, complex problems that don't default to a positive outcome without active care. Andrew did an amazing job laying the foundations, from deliberately choosing a non-profit corp
What is Amazon Simple Queue Service? - Amazon Simple Queue Service
Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and cost allocation tags. It provides a generic web services API that you can access using any programming language that the AWS SDK supports. Benefits of using
Using the circuit-breaker pattern with AWS Lambda extensions and Amazon DynamoDB | Amazon Web Services
AWS Compute Blog Using the circuit-breaker pattern with AWS Lambda extensions and Amazon DynamoDB This post is written by Alan Oberto Jimenez, Senior Cloud Application Architect, and Tobias Drees, Cloud Application Architect. Modern software systems frequently rely on remote calls to other systems across networks. When failures occur, they can cascade across multiple services causing service disru
Investigation of a Cross-regional Network Performance Issue
Hechao Li, Roger Cruz Cloud Networking TopologyNetflix operates a highly efficient cloud computing infrastructure that supports a wide array of applications essential for our SVOD (Subscription Video on Demand), live streaming and gaming services. Utilizing Amazon AWS, our infrastructure is hosted across multiple geographic regions worldwide. This global distribution allows our applications to del
Chaos Mesh 1.0: Chaos Engineering on Kubernetes Made Easier | Chaos Mesh
Chaos-Mesh-1.0 - Chaos-Engineering-on-Kubernetes-Made-Easier Today, we are proud to announce the general availability of Chaos Mesh 1.0, following its entry into CNCF as a sandbox project in July, 2020. Chaos Mesh 1.0 is a major milestone in the project’s development. After 10 months of effort within the open-source community, Chaos Mesh is now ready in terms of functionality, scalability, and eas
Everyone loves Unsolved Mysteries. There’s always someone who seems like the surefire culprit. There’s a clear motive, the perfect opportunity, and an incriminating footprint left behind. Yet, this is Unsolved Mysteries! It’s never that simple. Whether it’s a cryptic note behind the TV or a mysterious phone call from an unknown number at a critical moment, the pieces rarely fit together perfectly.
Learning distributed systems is tough. You need to go through a lot of academic papers, concepts, code review, before being able to have a global pictures. Thankfully, there is a lot of resources out there that can help you to get started. Here's a list of resources I used to learn distributed systems. I will keep this blogpost up-to-date with books, conferences, and so on. A distributed system is
Everything here is my opinion. I do not speak for your employer. Systems design explains the world: volume 1 "Systems design" is a branch of study that tries to find universal architectural patterns that are valid across disciplines. You might think that's not a possibility. Back in university, students used to tease the Systems Design Engineers, calling it "boxes and arrows" engineering. Not real
Kafka Listeners - Explained Published Aug 2, 2018 by in Apache Kafka, Docker, Advertised.listeners, Listeners, Aws, Ec2, KAFKA_ADVERTISED_LISTENERS at https://rmoff.net/2018/08/02/kafka-listeners-explained/ (This was cross-posted on the Confluent.io blog) This question comes up on StackOverflow and such places a lot, so here’s something to try and help. tl;dr : You need to set advertised.listeners
Use AWS Step Functions to Monitor Services Choreography | Amazon Web Services
AWS Architecture Blog Use AWS Step Functions to Monitor Services Choreography Organizations frequently need access to quick visual insight on the status of complex workflows. This involves collaboration across different systems. If your customer requires assistance on an order, you need an overview of the fulfillment process, including payment, inventory, dispatching, packaging, and delivery. If y
NoSQL Database Interview Questions for Experienced Candidates
Here is a list of common NoSQL interview questions with detailed answers for experienced NoSQL Database professionals. Through advanced queries about architecture, optimization, and real-world problem-solving in NoSQL environments, such interviews aim to gauge the candidate’s ability to understand the complexities of NoSQL databases in scalable, high-performance applications. In such interviews, o
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
In Search of an Understandable Consensus Algorithm
Twelve-factor app anno 2022 | Xenit
A Distributed Systems Reading List
How HTTP Keep-Alive can cause TCP race condition
Introduction | Hydro - Build for Every Scale
Innovating with Rust | Amazon Web Services
TEMPLE: Six Pillars of Observability
雰囲気で分散システム使ってるやついる!? - Qiita
An introduction to lockless algorithms [LWN.net]
暗号資産における取引の追跡困難性と匿名性：研究動向と課題
Vector databases (1): What makes each one different?
アンチウイルス製品では検知困難な「マルウェアフリー」の攻撃が増加 “ボディーガードのような守り”が防御の鍵に
Acton Programming Language
Edgar: Solving Mysteries Faster with Observability
Best resources to learn about data and distributed systems
Systems design explains the world: volume 1
Kafka Listeners - Explained