はてなブックマーク

What would a totally new search engine architecture look like? Who better than Julien Lemoine, Co-founder & CTO of Algolia, to describe what the future of search will look like. This is the first article in a series. Search engines, and more generally, information retrieval systems, play a central role in almost all of today’s technical stacks. Information retrieval started in the beginning of com

Now, let‘s get started. Linux buddy memory allocatorLinux uses the buddy algorithm as a page allocator, which is simple and efficient. Linux has made some extensions to the classic algorithm: Partitions' buddy allocatorPer-CPU pagesetGroup by migration typesThe Linux kernel uses node, zone, and page to describe physical memory. The partitions' buddy allocator focuses on a certain zone on a certain

Zoom scaled from 20 million to 300 million users virtually over night. What's incredible is from the outside they've shown little in the way of apparent growing pains, though on the inside it's a good bet a lot of craziness is going on. Sure, Zoom has made some design decisions that made sense as a small spunky startup that don't make a lot of sense as a defacto standard, but that's to be expected

Identify yourself with this website. All requests to log into this website are logged. Repeated failed login attempts will result in lock out. You are attempting to access a resource on this site which is restricted. Please login below. After re-logging in, you will be automatically directed to the page you were attempting to access.

AWS is the #1 cloud provider for open-source database hosting, and the go-to cloud for MySQL deployments. As organizations continue to migrate to the cloud, it’s important to get in front of performance issues, such as high latency, low throughput, and replication lag with higher distances between your users and cloud infrastructure. While many AWS users default to their managed database solution,

This is a guest post by Roger Jin, Software Architect at ButterCMS and co-author of Microservices for Startups. For a profession that stresses the importance of naming things well, we've done ourselves a disservice with microservices. The problem is that that there is nothing inherently "micro" about microservices. Some can be small, but size is relative and there's no standard of unit of measure

Original article available at https://habrahabr.ru/company/mailru/blog/323870/ I’d like to share with you an article based on my talk at Tarantool Meetup(the video is in Russian, though). It’s a short story of why Mamba, one of the biggest dating websites in the world and the largest one in Russia, started using Tarantool. Why did we decide to busy ourselves with MySQL-to-Tarantool replication? Fi

In this article, I want to share with you how I solved a very interesting problem of synchronizing data between IoT devices and a cloud application. I’ll start by outlining the general idea and the goals of my project. Then I’ll describe my implementation in greater detail. This is going to be a more technically advanced part, where I’ll be talking about the Contiki OS, databases, protocols and th

« Stuff The Internet Says On Scalability For September 30th, 2016 | Main | Sponsored Post: ScaleArc, Spotify, Aerospike, Scalyr, Gusto, VividCortex, MemSQL, InMemory.Net, Zohocorp » If you are Uber and you need to store the location data that is sent out every 30 seconds by both driver and rider apps, what do you do? That’s a lot of real-time data that needs to be used in real-time. Uber’s solutio

This is a guest repost from Baqend Tech on deploying and redeploying an Apache Storm cluster on top of Docker Swarm instead of deploying on VMs. It's an interesting topic because of the experience Wolfram Wingerath called it "a real joy", which is not a phrase you hear often in tech. Curious, I asked what made using containers such a good experience over using VMs? Here's his reply: When I wrote t

Today Twitter is creating and persisting 3,000 (200 GB) images per second. Even better, in 2015 Twitter was able to save $6 million due to improved media storage policies. It was not always so. Twitter in 2012 was primarily text based. A Hogwarts without all the cool moving pictures hanging on the wall. It’s now 2016 and Twitter has moved into to a media rich future. Twitter has made the transitio

If you can’t understand what’s in information then it’s going to be very difficult to organize it. This quote is from Jeff Dean, currently a Wizard, er, Fellow in Google’s Systems Infrastructure Group. It’s taken from his recent talk: Large-Scale Deep Learning for Intelligent Computer Systems. Since AlphaGo vs Lee Se-dol, the modern version of John Henry’s fatal race against a steam hammer, has ca

Hi, I'm Manu Mahajan and I'm a software engineer with Keen IO's Platform team. Over the past year I've focused on improving our query performance and scalability. I wanted to share some things we've learned from this experience in a series of posts. Today, I'll describe how we're working to guarantee consistent performance in a multi-tenant environment built on top of Apache Storm. tl;dr we were a

This is a guest post by Benjamin Manes, who did engineery things for Google and is now doing engineery things for a new load documentation startup, LoadDocs. Caching is a common approach for improving performance, yet most implementations use strictly classical techniques. In this article we will explore the modern methods used by Caffeine, an open-source Java caching library, that yield high hit

How do you scale a system from one user to more than 11 million users? Joel Williams, Amazon Web Services Solutions Architect, gives an excellent talk on just that subject: AWS re:Invent 2015 Scaling Up to Your First 10 Million Users. If you are an advanced AWS user this talk is not for you, but it’s a great way to get started if you are new to AWS, new to the cloud, or if you haven’t kept up with

This is a guest repost by Chris Ueland, creator of Scale Scale, with a creative high level view of the Netflix stack. As we research and dig deeper into scaling, we keep running into Netflix. They are very public with their stories. This post is a round up that we put together with Bryan’s help. We collected info from all over the internet. If you’d like to reach out with more info, we’ll append t

This is a guest repost from Calvin French-Owen, CTO/Co-Founder of Segment. In Segment’s early days, our infrastructure was pretty hacked together. We provisioned instances through the AWS UI, had a graveyard of unused AMIs, and configuration was implemented three different ways. As the business started taking off, we grew the size of the eng team and the complexity of our architecture. But working

For a long time now stateless services have been the royal road to scalability. Nearly every treatise on scalability declares statelessness as the best practices approved method for building scalable systems. A stateless architecture is easy to scale horizontally and only requires simple round-robin load balancing. What’s not to love? Perhaps the increased latency from the roundtrips to the databa

Google with justly earned pride recently announced: Today at the 2015 Open Network Summit, we are revealing for the first time the details of five generations of our in-house network technology. From Firehose, our first in-house datacenter network, ten years ago to our latest-generation Jupiter network, we’ve increased the capacity of a single datacenter network more than 100x. Our current generat

How do you program a computer with 10 terabytes of RAM in a single address space?  When the great Adrian Cockcroft was interviewed for Enterprise Initiatives Episode blog, that’s one of the answers he gave to the question of “What’s the next big thing?” Adrian says we are already taking big machines and running tiny little containers on them. He thinks another interesting workload is huge memory s