Netflix Conductor
In December 2016, we open sourced Netflix Conductor. We have been working hard since then to add more features, enhance the user interface for monitoring the workflows, and harden the system. We have seen an increase in usage, new use cases, feature requests, and lot of interest from the community. In this post, we will talk about a couple of major features we added to Conductor this quarter. IoC
NetflixOSS: Announcing Hollow
“If you can cache everything in a very efficient way, you can often change the game” We software engineers often face problems that require the dissemination of a dataset which doesn’t fit the label “big data”. Examples of this type of problem include: Product metadata on an ecommerce siteDocument metadata in a search engineMetadata about movies and TV shows on an Internet television networkWhen f
Vizceral Open Source
Previously we wrote about our traffic intuition tool, Flux. We have some announcements and updates to share about this project. First, we have renamed the project to Vizceral. More importantly, Vizceral is now open source! Open SourceVizceral transformed the way we understand and digest information about the state of traffic flowing into the Netflix control plane. We wanted to be able to intuit de
Meson: Workflow Orchestration for Netflix Recommendations
At Netflix, our goal is to predict what you want to watch before you watch it. To do this, we run a large number of machine learning (ML) workflows every day. In order to support the creation of these workflows and make efficient use of resources, we created Meson. Meson is a general purpose workflow orchestration and scheduling framework that we built to manage ML pipelines that execute workloads
Kafka Inside Keystone Pipeline
In summary, the Keystone pipeline is a unified event publishing, collection, and routing infrastructure for both batch and stream processing. We have two sets of Kafka clusters in Keystone pipeline: Fronting Kafka and Consumer Kafka. Fronting Kafka clusters are responsible for getting the messages from the producers which are virtually every application instance in Netflix. Their roles are data co
Stream-processing with Mantis
Back in January of 2014 we wrote about the need for better visibility into our complex operational environments: The core of the message in that post was about the need for fine-grained, contextual and scalable insights into the experiences of our customers and behaviors of our services. While our execution has evolved somewhat differently from our original vision, the underlying principles behind
Evolution of the Netflix Data Pipeline
Our new Keystone data pipeline went live in December of 2015. In this article, we talk about the evolution of Netflix’s data pipeline over the years. This is the first of a series of articles about the new Keystone data pipeline. Netflix is a data-driven company. Many business and product decisions are based on insights derived from data analysis. The charter of the data pipeline is to collect, ag
Linux Performance Analysis in 60,000 Milliseconds
You log in to a Linux server with a performance issue: what do you check in the first minute? At Netflix we have a massive EC2 Linux cloud, and numerous performance analysis tools to monitor and investigate its performance. These include Atlas for cloud-wide monitoring, and Vector for on-demand instance analysis. While those tools help us solve most issues, we sometimes need to login to an instanc
From Chaos to Control
by Leena Janardanan, Bruce Wobbe, Vilas Veeraraghavan IntroductionMerchandising Application Platform (MAP) was conceived as a middle-tier service that would handle real time requests for content discovery. MAP does this by aggregating data from disparate data sources and implementing common business logic into one distinct layer. This centralized layer helps provide common experiences across devic
Fenzo: OSS Scheduler for Apache Mesos Frameworks
Bringing Netflix to our millions of subscribers is no easy task. The product comprises dozens of services in our distributed environment, each of which is operating a critical component to the experience while constantly evolving with new functionality. Optimizing the launch of these services is essential for both the stability of the customer experience as well as overall performance and costs. T
Can Spark Streaming survive Chaos Monkey?
by Bharat Venkat, Prasanna Padmanabhan, Antony Arokiasamy, and Raju Uppalapati Netflix is a data-driven organization that places emphasis on the quality of data collected and processed. In our previous blog post, we highlighted our use cases for real-time stream processing in the context of online recommendations and data monitoring: With Spark Streaming as our choice of stream processor, we set o
Prana: A Sidecar for your Netflix PaaS based Applications and Services
“The Right Tool for the Right Job” It’s hard to argue against this time tested mantra. At Netflix, an overwhelming part of our applications and services have traditionally been implemented in Java. As our services and products evolved, we asked ourselves if Java was still the right choice for implementing these services/applications. While it’s still true that for most of our microservices Java is
Introducing Dynomite
Introduction & OverviewNetflix has long been a proponent of the microservices model. This model offers higher-availability, resiliency to failure and loose coupling. The downside to such an architecture is the potential for a latent user experience. Every time a customer loads up a homepage or starts to stream a movie, there are a number of microservices involved to complete that request. Most of
Optimizing the Netflix Streaming Experience with Data Science
by Nirmal Govind On January 16, 2007, Netflix started rolling out a new feature: members could now stream movies directly on their browser without having to wait for the red envelope in the mail. This event marked a substantial shift for Netflix and the entertainment industry. A lot has changed since then. Today, Netflix delivers over 1 billion hours of streaming per month to 48 million members in
Distributed Neural Networks with GPUs in the AWS Cloud
by Alex Chen, Justin Basilico, and Xavier Amatriain As we have described previously on this blog, at Netflix we are constantly innovating by looking for better ways to find the best movies and TV shows for our members. When a new algorithmic technique such as Deep Learning shows promising results in other domains (e.g. Image Recognition, Neuro-imaging, Language Models, and Speech Recognition), it
S3mper
In previous posts, we discussed how the Hadoop platform at Netflix leverages AWS’s S3 offering (read more here). In short, Netflix considers S3 the “source of truth” for all data warehousing. There are many attractive features that draw us to this service including: 99.999999999% durability, 99.99% availability, effectively infinite storage, versioning (data recovery), and ubiquitous access. In co
Introducing PigPen
It is our pleasure to release PigPen to the world today. PigPen is map-reduce for Clojure. It compiles to Apache Pig, but you don’t need to know much about Pig to use it. What is PigPen?A map-reduce language that looks and behaves like clojure.coreThe ability to write map-reduce queries as programs, not scriptsStrong support for unit tests and iterative developmentNote: If you are not familiar at
Announcing Suro
To make the best business and technical decisions, it is critical for Netflix to reliably collect application specific data in a timely fashion. At Netflix we deploy a fairly large number of AWS EC2 instances that host our web services and applications. They collectively emit more than 1.5 million events per second during peak hours, or around 80 billion events per day. The events could be log mes
Netflix Open Source Software Cloud Prize Winners
We launched the Netflix Open Source Software Cloud Prize in March 2013 and it got a lot of attention in the press and blogosphere. Six months later we closed the contest, took a good look at the entrants, picked the best as nominees, and the panel of distinguished judges decided the winners in each category. The final winners were announced during Werner Vogel’s keynote at AWS Re:Invent in on Nove
Scryer: Netflix’s Predictive Auto Scaling Engine
To deliver the best possible experience to Netflix customers around the world, it is critical for us to maintain a robust, scalable, and resilient system. That is why we have built (and open sourced) applications ranging from Hystrix to Chaos Monkey. All of these tools better enable us to prevent or minimize outages, respond effectively to outages, and/or anticipate the kinds of operational gaps t
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
