はてなブックマーク

2022-10-27 I’m proud to announce, after nearly a year of work, Sidekiq 7.0 is now available. Sidekiq is the most popular background job system for Ruby, used by thousands of companies around the world. This release is our biggest, most splendiforous release ever! What’s New? Metrics One thing I know: everybody loves big beautiful graphs! Sidekiq 7.0 has a major new feature for tracking and visuali

2022-07-14 Einhorn is a Ruby gem which provides the ability to manage multiple forked processes of Unix services. Stripe originally built Einhorn to manage their internal collection of services in production but stopped actively maintaining the gem several years ago. I recently asked for and was granted ownership of the gem so that I could provide maintenance going forward. This blog post is meant

2022-01-17 It’s hard for me to believe these words but I pushed Sidekiq’s first commit on Jan 16th, 2012. Ten years ago. The public announcement. One month later. One quarter later. Some context for those new to this blog: Sidekiq is the most popular background job system for the Ruby programming language. Every application has tasks which are important: send an email, charge a credit card, reserv

2019-09-03 I’m happy to announce that Sidekiq 6.0, Sidekiq Pro 5.0 and Sidekiq Enterprise 2.0 are now generally available after nine months of work by various contributors! 🎉🎂 No More Daemonizing Sidekiq 6.0 no longer offers the ability to run as an circa-1990s init.d-style daemon by removing the logfile, pidfile and daemonize command line flags. For a decade, all major Unix systems have offered

2018-04-25 MRI, the standard Ruby interpreter, has a serious problem with memory bloat in large Rails apps. It’s quite common for me to see Sidekiq processes which are 1-2GB in RSS or even larger! It turns out that a large part of this memory usage is due to memory fragmentation: MRI uses the OS’s memory allocator by default (on Linux, almost always GNU glibc), which seems to work poorly with Ruby

2018-02-28 Software performance optimization is simple: find a way to do less. Ruby has a reputation for being slow and, while that impression is a decade out of date, one of the leading offenders has been the garbage collector. This leads to the question: can we speed up Ruby by creating less garbage? Absolutely! A String Primer Ruby has an unfortunate default semantic that all Strings are mutabl

2018-01-08 Faktory is my brand-new background job system for every programming language. If you want to learn more, see the intro. It’s been three months since the initial launch in October and I’ve heard of several successful rollouts to production. If you were reluctant to try out Faktory before, it’s time to start looking into it because the biggest release of Faktory yet is now available to al

Several customers have been asking for more metrics to track Sidekiq internals: when is a job dropped due to uniqueness or expiration? I've had to put off the work while working on Faktory but recently I took a week off to focus on Sidekiq and implement this feature. New and Improved Sidekiq Pro 3.6 has a brand new Statsd metrics subsystem. Statsd is a de facto universal standard for metrics in op

2017-11-13 When I unveiled my new background job system Faktory three weeks ago, I didn’t have an easy path for people to install and try Faktory themselves. Now there is; let’s get it running! Homebrew If you are running macOS, it’s installable with Homebrew: > brew tap contribsys/faktory > brew install faktory ...two minutes pass... > faktory Now open your browser to http://localhost:7420 Docker

2017-10-23 Whether sending an email, adding an item to your cart or tracking an impression, modern businesses can be modeled as a series of business transactions. I believe that a good background job framework is the best way to execute these transactions and scale your business applications across many machines. A good background job framework gives you: a simple API for defining and calling jobs

2017-10-24 Yesterday I explained how a good background job framework is critical to scaling business applications. Today, meet Faktory. I’ve spent the last six months building Faktory, a new background job system which aims to bring the best practices developed over the last five years in Sidekiq to every programming language. The Basics Faktory is a server daemon which provides a simple API to pr

2017-09-29 Do you hate getting your inbox filled with errors you know you can ignore? Here’s one cool trick to make them disappear. If you are writing a background job, it is best practice to have that job retry in the case of unexpected errors. Networks can be flaky, code can be buggy, data can be sketchy – as they say, “stuff happens”. Sidekiq’s default policy for jobs and retries is simple: if

2015-11-23 Every month or so we hear of another high profile open source developer who flips the table and rages about open source because of burnout, most recently Ian Cordasco and Ryan Bigg. Does this sound familiar? Spend many hours writing software. Spend many hours supporting users and solving their problems. Make $0. If you were a freelancer and did that with clients, you’d be: Overwhelmed w

2016-05-25 Sidekiq is a popular job framework for Ruby. Now we’re bringing it to Crystal! Why Crystal? I wanted to use a language that was a good complement to Ruby. Its syntax is similar enough to Ruby that I can reuse a lot of code but it adds a huge leap in performance. In summary: very similiar syntax to Ruby at least 3-5x faster than Ruby 2.3 on most code at least 3x smaller in memory footpri

2016-02-09 This post talks about Ruby but it’s true of every language community: Python, JavaScript, Java, etc. The scourge of dependencies spares no one. This is a dependency visualization of every Rails app I’ve ever used. Does any of this sound familiar: Gemfile with 100s of entries. Test gems loading in production. Each Rails process takes 100s of megabytes of RAM. The Rubygems system is comme

2015-12-14 Multithreaded code is hard to write and even harder to test. Since much of my work is dedicated to making Ruby threading easier for my users and customers, I thought some might be interested in the patterns I’ve developed to make multithreaded code as simple and testable as possible. Separate Threading from Work If you can’t test a big block of code, break it into a set of smaller testa

2015-10-14 Sidekiq has a reputation for being much faster than its competition but there’s always room for improvement. I recently rewrote its internals and made it six times faster. Here’s how! It’s been quite a while since I’ve touched Sidekiq’s core design. That was intentional: for the last year Sidekiq has stabilized and become reliable infrastructure that Ruby developers can trust when build

2015-09-24 Would you stuff all of your data into one database table? That’s crazy, Mike, don’t be silly! What if I told you most people do just that with Redis? Redis users often have several distinct datasets in Redis: long-lived transactional data, background job queues, ephemeral cached data, etc. At the same time I see lots of people using Redis in the most naive way possible: put everything i

2013-02-23 A user recently ran Sidekiq with Ruby 2.0 and found that the signal handling did not work well at all. Ctrl-C and other signals resulted in some ominous stack traces. It turns out that Ruby 2.0 locks down what you can do in a signal handler in order to prevent unsafe or possibly non-deterministic behavior. You can’t take a Mutex within a signal handler anymore as this could result in a

2014-12-11 Go 1.4 is out! I won’t reiterate the changes, read the release notes for that. But it’s one thing to read that things have improved – what if I could show you the improvement? These snapshots are from an upcoming feature for Inspeqtor Pro: real-time memory and GC monitoring for any Go-based daemon. They show the first 3 minutes of Inspeqtor Pro itself starting up. After 1 minute, it sen