はてなブックマーク

This post describes WePay’s highly available MySQL architecture, and how we achieve short outage times during failures. WePay uses a variety of relational database management systems (RDBMS) and noSQL databases. MySQL remains the main RDBMS that completes all critical operations. The site and all API transactions require MySQL access. We run multiple MySQL clusters serving different sets of servic

Change data capture has been around for a while, but some recent developments in technology have given it new life. Notably, using Kafka as a backbone to stream your database data in realtime has become increasingly common. If you’re wondering why you might want to stream database changes into Kafka, I highly suggest reading The Hardest Part About Microservices: Your Data. At WePay, we wanted to i

NOTE: We recently gave an Airflow at WePay talk to the Bay Area Airflow meetup group. The video and slides are both available. Our last post provided an overview of WePay’s data warehouse. In this post, we’ll be diving into how we run Airflow as part of the ETL pipeline. Introduction Airflow is a workflow scheduler. It lets you define a series of tasks (chunks of code, queries, etc) that can be st