Data Mesh — A Data Movement and Processing Platform @ Netflix
By Bo Lei, Guilherme Pires, James Shao, Kasturi Chatterjee, Sujay Jain, Vlad Sydorenko BackgroundRealtime processing technologies (A.K.A stream processing) is one of the key factors that enable Netflix to maintain its leading position in the competition of entertaining our users. Our previous generation of streaming pipeline solution Keystone has a proven track record of serving multiple of our ke
Data Engineers of Netflix — Interview with Kevin Wylie
This post is part of our “Data Engineers of Netflix” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Kevin Wylie is a Data Engineer on the Content Data Science and Engineering team. In this post, Kevin talks about his extensive experience in content analytics at Netflix since joining more than 10 years ago. Kevin grew up in the Washington, DC area
The Netflix Cosmos Platform
by Frank San Miguel on behalf of the Cosmos team IntroductionCosmos is a computing platform that combines the best aspects of microservices with asynchronous workflows and serverless functions. Its sweet spot is applications that involve resource-intensive algorithms coordinated via complex, hierarchical workflows that last anywhere from minutes to years. It supports both high throughput services
How Netflix Scales its API with GraphQL Federation (Part 2)
In our previous post and QConPlus talk, we discussed GraphQL Federation as a solution for distributing our GraphQL schema and implementation. In this post, we shift our attention to what is needed to run a federated GraphQL platform successfully — from our journey implementing it to lessons learned. Our Journey so FarOver the past year, we’ve implemented the core infrastructure pieces necessary fo
Supporting content decision makers with machine learning
by Melody Dye*, Chaitanya Ekanadham*, Avneesh Saluja*, Ashish Rastogi * contributed equally Netflix is pioneering content creation at an unprecedented scale. Our catalog of thousands of films and series caters to 195M+ members in over 190 countries who span a broad and diverse range of tastes. Content, marketing, and studio production executives make the key decisions that aspire to maximize each
Keeping Netflix Reliable Using Prioritized Load Shedding
By Manuel Correa, Arthur Gonigberg, and Daniel West Getting stuck in traffic is one of the most frustrating experiences for drivers around the world. Everyone slows to a crawl, sometimes for a minor issue or sometimes for no reason at all. As engineers at Netflix, we are constantly reevaluating how to redesign traffic management. What if we knew the urgency of each traveler and could selectively r
Building Netflix’s Distributed Tracing Infrastructure
“@Netflixhelps Why doesn’t Tiger King play on my phone?” — a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue — which is difficult when troubleshooting distributed systems. Investigating a video streaming failure consists of inspecting all aspects of a member account. In our previous blog post we introduced Edgar, our t
Telltale: Netflix Application Monitoring Simplified
By Andrei Ushakov, Seth Katz, Janak Ramachandran, Jeff Butsch, Peter Lau, Ram Vaithilingam, and Greg Burrell Our Telltale VisionAn alert fires and you get paged in the middle of the night. A metric crossed a threshold. You’re half awake and wondering, “Is there really a problem or is this just an alert that needs tuning? When was the last time somebody adjusted our alert thresholds? Maybe it’s due
Analytics at Netflix: Who we are and what we do
Across nearly every industry, there is recognition that data analytics is key to driving informed business decision-making. But there is far less agreement on what that term “data analytics” actually means — or what to call the people responsible for the work. Even within Netflix, we have many groups that do some form of data analysis, including business strategy and consumer insights. But here we
Unbundling Data Science Workflows with Metaflow and AWS Step Functions
by David Berg, Ravi Kiran Chirravuri, Romain Cledat, Jason Ge, Savin Goyal, Ferras Hamad, Ville Tuulos tl;dr Today, we are releasing a new job scheduler integration with AWS Step Functions. This integration allows the users of Metaflow to schedule their production workflows using a highly available, scalable, maintenance-free service, without any changes in their existing Metaflow code. The idea o
ML Platform Meetup: Infra for Contextual Bandits and Reinforcement Learning
Infrastructure for Contextual Bandits and Reinforcement Learning — theme of the ML Platform meetup hosted at Netflix, Los Gatos on Sep 12, 2019. Contextual and Multi-armed Bandits enable faster and adaptive alternatives to traditional A/B Testing. They enable rapid learning and better decision-making for product rollouts. Broadly speaking, these approaches can be seen as a stepping stone to full-o
Open Sourcing Mantis: A Platform For Building Cost-Effective, Realtime, Operations-Focused…
By Cody Rioux, Daniel Jacobson, Jeff Chao, Neeraj Joshi, Nick Mahilani, Piyush Goyal, Prashanth Ramdas, Zhenzhong Xu Today we’re excited to announce that we’re open sourcing Mantis, a platform that helps Netflix engineers better understand the behavior of their applications to ensure the highest quality experience for our members. We believe the challenges we face here at Netflix are not necessari
Reimagining Experimentation Analysis at Netflix
Toby Mao, Sri Sri Perangur, Colin McFarland Another day, another custom script to analyze an A/B test. Maybe you’ve done this before and have an old script lying around. If it’s new, it’s probably going to take some time to set up, right? Not at Netflix. ABlaze: The standard view of analyses in the XP UISuppose you’re running a new video encoding test and theorize that the two new encodes should r
GraphQL Search Indexing
by Artem Shtatnov and Ravi Srinivas Ranganathan Almost a year ago we described our learnings from adopting GraphQL on the Netflix Marketing Tech team. We have a lot more to share since then! There are plenty of existing resources describing how to express a search query in GraphQL and paginate the results. This post looks at the other side of search: how to index data and make it searchable. Speci
Open-Sourcing riskquant, a library for quantifying risk
Netflix has a program in our Information Security department for quantifying the risk of deliberate (attacker-driven) and accidental losses. This program started on the Detection Engineering team with a home-grown Python library called riskquant, which we’ve released as open source for you to use (and contribute to). Since that library was written, we have hired two amazing full-time Risk Engineer
Keeping Customers Streaming — The Centralized Site Reliability Practice at Netflix
By Hank Jacobs, Senior Site Reliability Engineer on CORE We’re privileged to be in the business of bringing joy to our customers at Netflix. Whether it’s a compelling new series or an innovative product feature, we strive to provide a best-in-class service that people love and can enjoy anytime, anywhere. A key underpinning to keeping our customers happy and streaming is a strong focus on reliabil
Byte Down: Making Netflix’s Data Infrastructure Cost-Effective
By Torio Risianto, Bhargavi Reddy, Tanvi Sahni, Andrew Park Background on data efficiencyAt Netflix, we invest heavily in our data infrastructure which is composed of dozens of data platforms, hundreds of data producers and consumers, and petabytes of data. At many other organizations, an effective way to manage data infrastructure costs is to set budgets and other heavy guardrails to limit spendi
Towards true continuous integration
For the past 8 years, Netflix has been building and evolving a robust microservice architecture in AWS. Throughout this evolution, we learned how to build reliable, performant services in AWS. Our microservice architecture decouples engineering teams from each other, allowing them to build, test and deploy their services as often as they want. This flexibility enables teams to maximize their deliv
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Data Engineers of Netflix — Interview with Dhevi Rajendran
Netflix at MIT CODE 2020