DBLog: A Generic Change-Data-Capture Framework
Andreas Andreakis, Ioannis Papapanagiotou OverviewChange-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional
Implementing the Netflix Media Database – Netflix TechBlog – Medium
In the previous blog posts in this series, we introduced the Netflix Media DataBase (NMDB) and its salient “Media Document” data model. In this post we will provide details of the NMDB system architecture beginning with the system requirements — these will serve as the necessary motivation for the architectural choices we made. A fundamental requirement for any lasting data system is that it shoul
Netflix at AWS re:Invent 2018 – Netflix TechBlog – Medium
by Shaun Blackburn AWS re:Invent is back in Las Vegas this week! Many Netflix engineers and leaders will be among the 40,000 attending the conference to connect with fellow cloud and OSS enthusiasts. You can find us at our booth on the expo floor, speaking on a variety of subjects, and at meetups and events around the re:Invent campus. We have listed all our talks below to make it easy to hear wha
Lumen: Custom, Self-Service Dashboarding For Netflix
Netflix generates a lot of data. One of the ways that we gain useful insights is by visualizing that data in dashboards which allow us to comprehend large amounts of information quickly. This is particularly important when operational issues arise as our engineers need to be able to quickly diagnose problem areas and work to correct them. Operational issues, however, are just one potential use cas
Multi-Cloud Continuous Delivery with Spinnaker report now available.
by Emily Burns, Asher Feldman, Rob Fletcher, Tomas Lin, Justin Reynolds, Chris Sanden and Rob Zienert We’re pleased to announce the release of our O’Reilly report, Continuous Delivery with Spinnaker. The report is available to download for free on the Spinnaker website. ( Pdf | Epub | Mobi ) About The ReportAt Netflix, we’ve built and use Spinnaker as a platform for continuous integration and deli
Titus, the Netflix container management platform, is now open source
by Amit Joshi, Andrew Leung, Corin Dwyer, Fabio Kung, Sargun Dhillon, Tomasz Bak, Andrew Spyker, Tim Bozarth Today, we are open-sourcing Titus, our container management platform. Titus powers critical aspects of the Netflix business, from video streaming, recommendations and machine learning, big data, content encoding, studio technology, internal engineering tools, and other Netflix workloads. Ti
Serving 100 Gbps from an Open Connect Appliance
In the summer of 2015, the Netflix Open Connect CDN team decided to take on an ambitious project. The goal was to leverage the new 100GbE network interface technology just coming to market in order to be able to serve at 100 Gbps from a single FreeBSD-based Open Connect Appliance (OCA) using NVM Express (NVMe)-based storage. At the time, the bulk of our flash storage-based appliances were close to
Simone - A Distributed Simulation Service
By Satyajit Thadeshwar, Mayank Agarwal, Sangeeta Narayanan & Kevin Lew Hundreds of models of smart TVs, game consoles, mobile devices, and other video streaming devices get shipped with a Netflix app pre-installed. Before shipping these devices, manufacturers need to have the app certified on their device firmware. Certification involves running a series of tests that validate the behavior of the
Spinnaker Orchestration
Author: Rob Fletcher When the Spinnaker project first started more than two years ago we implemented Orca — Spinnaker’s orchestration engine µservice — using Spring Batch. It wasn’t an entirely unreasonable fit at the time. It gave us atomic, compartmentalized units of work (tasks in a Spinnaker pipeline), retry semantics, the ability to define branching and joining workflows, listeners that could
Automation as a Service — Introducing Scriptflask
Authors : Fazal Allanabanda and Vilas Veeraraghavan IntroductionLess than a year ago, we shared our testing wins and the challenges that lie ahead while testing High Impact Titles (HITs) globally. All this while ensuring the pace of innovation at Netflix does not slow down. Since then, we have made significant enhancements to our automation. So in this post, we wanted to talk about our evolution s
The Evolution of Container Usage at Netflix
Containers are already adding value to our proven globally available cloud platform based on Amazon EC2 virtual machines. We’ve shared pieces of Netflix’s container story in the past (video, slides), but this blog post will discuss containers at Netflix in depth. As part of this story, we will cover Titus: Netflix’s infrastructural foundation for container based applications. Titus provides Netfli
Netflix Conductor: A microservices orchestrator
Conductor is a microservices orchestration engine - https://netflix.github.io/conductor/ Why not peer to peer choreography?With peer to peer task choreography, we found it was harder to scale with growing business needs and complexities. Pub/sub model worked for simplest of the flows, but quickly highlighted some of the issues associated with the approach: Process flows are “embedded” within the c
Engineering Trade-Offs and The Netflix API Re-Architecture
Netflix’s engineering culture is predicated on Freedom & Responsibility, the idea that everyone (and every team) at Netflix is entrusted with a core responsibility. Within that framework they are free to operate with freedom to satisfy their mission. Accordingly, teams are generally responsible for all aspects of their systems, ranging from design, architecture, development, deployments, and opera
Distributed delay queues based on Dynomite
Netflix’s Content Platform Engineering runs a number of business processes which are driven by asynchronous orchestration of micro-services based tasks, and queues form an integral part of the orchestration layer amongst these services. Few examples of these processes are: IMF based content ingest from our partnersProcess of setting up new titles within NetflixContent Ingest, encode and deployment
Building fast.com
On our company blog in May, we introduced fast.com, our new internet speed test. The idea behind fast.com is to provide a quick and simple way for any internet user to test their current internet speed, whether they are a Netflix member or not. Since fast.com was released, millions of internet users around the world have run the test. We have seen a lot of interest in the site and questions about
Introducing Winston
Netflix is a collection of microservices that all come together to enable the product you have come to love. Operating these micro services is also distributed across the owning teams and their engineers. We do not run a central operations team managing these individual services for availability. What we do instead is to invest in tools that help Netflix engineers operate their services for high a
Vizceral Open Source
Previously we wrote about our traffic intuition tool, Flux. We have some announcements and updates to share about this project. First, we have renamed the project to Vizceral. More importantly, Vizceral is now open source! Open SourceVizceral transformed the way we understand and digest information about the state of traffic flowing into the Netflix control plane. We wanted to be able to intuit de
Distributed Resource Scheduling with Apache Mesos
Netflix uses Apache Mesos to run a mix of batch, stream processing, and service style workloads. For over two years, we have seen an increased usage for a variety of use cases including real time anomaly detection, training and model building batch jobs, machine learning orchestration, and Node.js based microservices. The recent release of Apache Mesos 1.0 represents maturity of the technology tha
Netflix Billing Migration to AWS
The journey to the cloud at Netflix began in 2008, and after seven years of diligent effort, we have finally completed… For a company, its billing solution is its financial lifeline, while at the same time, it is a visible representation of a company’s attitude towards its customers. A great customer experience is one of Netflix’s core values. Considering the sensitive nature of Billing for its di
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Netflix at AWS re:Invent 2016