6 Lessons from Dropbox - One Million Files Saved Every 15 minutes - High Scalability -
« Twitter by the Numbers - 460,000 New Accounts and 140 Million Tweets Per Day | Main | Google and Netflix Strategy: Use Partial Responses to Reduce Request Sizes » Dropbox saves one million files every 15 minutes, more tweets than even Twitterers tweet. That mind blowing statistic was revealed by Rian Hunter, a Dropbox Engineer, in his presentation How Dropbox Did It and How Python Helped at PyC
7 Design Patterns for Almost-infinite Scalability - High Scalability -
Good article from manageability.com summarizing design patterns from Pat Helland's amazing paper Life beyond Distributed Transactions: an Apostate's Opinion. Entities are uniquely identified - each entity which represents disjoint data (i.e. no overlap of data between entities) should have a unique key.Multiple disjoint scopes of transactional serializability - in other words there are these 'enti
The Four Meta Secrets of Scaling at Facebook - High Scalability -
Aditya Agarwal, Director of Engineering at Facebook, gave an excellent Scale at Facebook talk that covers their architecture, but the talk is really more about how to scale an organization by preserving the best parts of its culture. The key take home of the talk is: You can get the code right, you can get the products right, but you need to get the culture right first. If you don't get the cultur
Twitter’s Plan to Analyze 100 Billion Tweets - High Scalability -
If Twitter is the “nervous system of the web” as some people think, then what is the brain that makes sense of all those signals (tweets) from the nervous system? That brain is the Twitter Analytics System and Kevin Weil, as Analytics Lead at Twitter, is the homunculus within in charge of figuring out what those over 100 billion tweets (approximately the number of neurons in the human brain) mea
Product: Resque - GitHub's Distrubuted Job Queue - High Scalability -
Queuing work for processing in the background is a time tested scalability strategy. Queuing also happens to be one of those much needed tools where it easy enough to forge for your own that we see a lot of different versions made. Resque is GitHub's take on a job queue and they've used it to process million and millions of jobs so far. What is Resque? Redis-backed library for creating background
13 Scalability Best Practices - High Scalability -
AFK Partners has release what they feel are the Best Practices for Scalability: Asynchronous - Use asynchronous communication when possible.Swim Lanes – Create fault isolated “swim lanes” of hardware by customer segmentation.Cache - Make use of cache at multiple layers.Monitoring - Understand your application’s performance from a customer’s perspective.Replication - Replicate databases for recover
Neo4j - a Graph Database that Kicks Buttox | High Scalability
Update: Social networks in the database: using a graph database. A nice post on representing, traversing, and performing other common social network operations using a graph database. If you are Digg or LinkedIn you can build your own speedy graph database to represent your complex social network relationships. For those of more modest means Neo4j, a graph database, is a good alternative. A graph
Dealing with multi-partition transactions in a distributed KV solution - High Scalability -
I've been getting asked about this a lot lately so I figured I'd just blog about it. Products like WebSphere eXtreme Scale work by taking a dataset, partitioning it using a key and then assigning those partitions to a number of JVMs. Each partition usually has a primary and a replica. These 'shards' are assigned to JVMs. A transactional application typically interacts with the data on a single par
Scaling Digg and Other Web Applications - High Scalability -
Joe Stump, Lead Architect at Digg, gave this presentation at the Web 2.0 Expo. I couldn't find the actual presentation, but fortunately Kris Jordan took some great notes. That's how key moments in history are accidentally captured forever. Joe was also kind enough to respond to my email questions with a phone call. In this first part of the post Joe shares some timeless wisdom that you may or may
Strategy: Flickr - Do the Essential Work Up-front and Queue the Rest - High Scalability -
This strategy is stated perfectly by Flickr's Myles Grant: The Flickr engineering team is obsessed with making pages load as quickly as possible. To that end, we’re refactoring large amounts of our code to do only the essential work up front, and rely on our queuing system to do the rest. Flickr uses a queuing system to process 11 million tasks a day. Leslie Michael Orchard also does a great job e
Sharding the Hibernate Way - High Scalability -
Update: A very nice JavaWorld podcast interview with Google engineer Max Ross on Hibernate Shards. Max defines Hibernate Shards (horizontal partitioning), how it works (pretty well), virtual shards (don't ask), what they need to do in the future (query, replication, operational tools), and how it relates to Google AppEngine (not much). To scale you are supposed to partition your data. Sounds good,
How I Learned to Stop Worrying and Love Using a Lot of Disk Space to Scale - High Scalability -
Update 3: ReadWriteWeb says Google App Engine Announces New Pricing Plans, APIs, Open Access. Pricing is specified but I'm not sure what to make of it yet. An image manipulation library is added (thus the need to pay for more CPU :-) and memcached support has been added. Memcached will help resolve the can't write for every read problem that pops up when keeping counters. Update 2: onGWT.com threw
20 New Rules for Faster Web Pages - High Scalability -
Update: Nice explanation in The importance of bandwidth versus latency of how long latencies cause cascading delays in resource loading. Doloto tries to optimize how resources are loaded. Twenty new rules have been added to the original 14 rules for sizzling web performance. Part of scalability is worrying about performance too. The front-end is where 80-90% of end-user response time is spent and
Friends for Sale Architecture - A 300 Million Page View/Month Facebook RoR App - High Scalability -
Update: Jake in Does Django really scale better than Rails? thinks apps like FFS shouldn't need so much hardware to scale. In a short three months Friends for Sale (think Hot-or-Not with a market economy) grew to become a top 10 Facebook application handling 200 gorgeous requests per second and a stunning 300 million page views a month. They did all this using Ruby on Rails, two part time develope
Product: The Spread Toolkit - High Scalability -
Complex applications coordinating work across a lot of machines often need a highly performing fault tolerant message layer. Though a blast to write, it's probably a better use of your time to use an off the shelf solution. And that's where Spread comes in. Flickr, for example, uses Spread to create real-time event feeds from their web server logs. What exactly is Spread? From the Spread website:
Is there a difference between partitioning and federation and sharding? - High Scalability -
Unlike Theo Schlossnagle, author of Scalable Internet Architectures, I am not a stickler for semantics because I have an unswerving faith in the ultimate unknowability of the world as experienced by others. That's why it is Theo who bravely tackles the differences in his informative blog post Partitioning vs. Federation vs. Sharding. Royans Tharakan also talks about it on his blog. Is there a diff
An Unorthodox Approach to Database Design : The Coming of the Shard - High Scalability -
Update 4: Why you don’t want to shard. by Morgon on the MySQL Performance Blog. Optimize everything else first, and then if performance still isn’t good enough, it’s time to take a very bitter medicine. Update 3: Building Scalable Databases: Pros and Cons of Various Database Sharding Schemes by Dare Obasanjo. Excellent discussion of why and when you would choose a sharding architecture, how to sha
High Scalability | Building bigger, faster, more reliable websites.
Brief History of Scaling Uber This blog post was written by Josh Clemm, Senior Director of Engineering at Uber Eats. This is a repost from his LinkedIn article, approved by the author. On a cold evening in Paris in 2008, Travis Kalanick and Garrett Camp couldn't get a cab. That's when Behind AWS S3’s Massive Scale This is a guest article by Stanislav Kozlovski, an Apache Kafka Committer. If you wo
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Strategy: Disk Backup for Speed, Tape Backup to Save Your Bacon, Just Ask Google - High Scalability -
Manage Downtime Risk by Connecting Multiple Data Centers into a Secure Virtual LAN - High Scalability -