Reverse Searching Netflix’s Federated Graph
By Ricky Gardiner, Alex Hutter, and Katie Lefevre Since our previous posts regarding Content Engineering’s role in enabling search functionality within Netflix’s federated graph (the first post, where we identify the issue and elaborate on the indexing architecture, and the second post, where we detail how we facilitate querying) there have been significant developments. We’ve opened up Studio Sea
Application data caching using SSDs
With the global expansion of Netflix earlier this year came the global expansion of data. After the Active-Active project and now with the N+1 architecture, the latest personalization data needs to be everywhere at all times to serve any member from any region. Caching plays a critical role in the persistence story for member personalization as detailed in this earlier blog post: There are two pri
Streaming SQL in Data Mesh
Democratizing Stream Processing @ Netflix By Guil Pires, Mark Cho, Mingliang Liu, Sujay Jain Data powers much of what we do at Netflix. On the Data Platform team, we build the infrastructure used across the company to process data at scale. In our last blog post, we introduced “Data Mesh” — A Data Movement and Processing Platform. When a user wants to leverage Data Mesh to move and transform data,
Seeing through hardware counters: a journey to threefold performance increase | by Netflix Technology Blog | Nov, 2022 | Netflix TechBlog
By Vadim Filanovsky and Harshad Sane In one of our previous blogposts, A Microscope on Microservices we outlined three broad domains of observability (or “levels of magnification,” as we referred to them) — Fleet-wide, Microservice and Instance. We described the tools and techniques we use to gain insight within each domain. There is, however, a class of problems that requires an even stronger lev
Towards a Reliable Device Management Platform
By Benson Ma, Alok Ahuja IntroductionAt Netflix, hundreds of different device types, from streaming sticks to smart TVs, are tested every day through automation to ensure that new software releases continue to deliver the quality of the Netflix experience that our customers enjoy. In addition, Netflix continuously works with its partners (such as Roku, Samsung, LG, Amazon) to port the Netflix SDK
How Netflix uses eBPF flow logs at scale for network insight
By Alok Tiagi, Hariharan Ananthakrishnan, Ivan Porto Carrero and Keerti Lakshminarayan Netflix has developed a network observability sidecar called Flow Exporter that uses eBPF tracepoints to capture TCP flows at near real time. At much less than 1% of CPU and memory on the instance, this highly performant sidecar provides flow data at scale for network insight. ChallengesThe cloud network infrast
How Netflix Scales its API with GraphQL Federation (Part 2)
In our previous post and QConPlus talk, we discussed GraphQL Federation as a solution for distributing our GraphQL schema and implementation. In this post, we shift our attention to what is needed to run a federated GraphQL platform successfully — from our journey implementing it to lessons learned. Our Journey so FarOver the past year, we’ve implemented the core infrastructure pieces necessary fo
How Netflix Scales its API with GraphQL Federation (Part 1)
Netflix is known for its loosely coupled and highly scalable microservice architecture. Independent services allow for evolving at different paces and scaling independently. Yet they add complexity for use cases that span multiple services. Rather than exposing 100s of microservices to UI developers, Netflix offers a unified API aggregation layer at the edge. UI developers love the simplicity of w
DBLog: A Generic Change-Data-Capture Framework
Andreas Andreakis, Ioannis Papapanagiotou OverviewChange-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional
Open-Sourcing Metaflow, a Human-Centric Framework for Data Science
by David Berg, Ravi Kiran Chirravuri, Romain Cledat, Savin Goyal, Ferras Hamad, Ville Tuulos tl;dr Metaflow is now open-source! Get started at metaflow.org. Netflix applies data science to hundreds of use cases across the company, including optimizing content delivery and video encoding. Data scientists at Netflix relish our culture that empowers them to work autonomously and use their judgment to
Re-Architecting the Video Gatekeeper
This is the story about how the Content Setup Engineering team used Hollow, a Netflix OSS technology, to re-architect and simplify an essential component in our content pipeline — delivering a large amount of business value in the process. The ContextEach movie and show on the Netflix service is carefully curated to ensure an optimal viewing experience. The team responsible for this curation is Ti
Python at Netflix
By Pythonistas at Netflix, coordinated by Amjith Ramanujam and edited by Ellen Livengood As many of us prepare to go to PyCon, we wanted to share a sampling of how Python is used at Netflix. We use Python through the full content lifecycle, from deciding which content to fund all the way to operating the CDN that serves the final video to 148 million members. We use and contribute to many open-sou
Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…
By: Di Lin, Girish Lingappa, Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question — “Can I run a check myself to understand what data is behind this metric?” Now, imagine yourself in the role of a software engineer responsible for a micro-service which publishes dat
Using Presto in our Big Data Platform on AWS
by Eva Tse, Zhenxiao Luo, Nezih Yigitbasi @ Big Data Platform team At Netflix, the Big Data Platform team is responsible for building a reliable data analytics platform shared across the whole company. In general, Netflix product decisions are very data driven. So we play a big role in helping different teams to gain product and consumer insights from a multi-petabyte scale data warehouse (DW). Th
Rethinking Netflix’s Edge Load Balancing – Netflix Technology Blog – Medium
By Mike Smith, Netflix Cloud Gateway We briefly touched on some of the load balancing improvements we’ve recently been making in our Open Sourcing Zuul 2 post. In this post, we’ll go into more detail on the whys, hows and results of that work. GoalsOn the Netflix Cloud Gateway team we are always working to help systems reduce errors, gain higher availability, and improve Netflix’s resilience to fa
Evolution of Application Data Caching : From RAM to SSD
In 2013, we introduced EVCache a distributed in-memory caching solution based on memcached that offers low-latency, high-reliability caching and storage. It is well integrated with AWS and EC2, a Netflix OSS project, and in many occasions it is termed as “the hidden microservice”. Since then, EVCache has become a fundamental tier-0 service storing petabytes of data and hundred of billions of items
Using Machine Learning to Improve Streaming Quality at Netflix
by Chaitanya Ekanadham One of the common questions we get asked is: “Why do we need machine learning to improve streaming quality?” This is a really important question, especially given the recent hype around machine learning and AI which can lead to instances where we have a “solution in search of a problem.” In this blog post, we describe some of the technical challenges we face for video stream
Introducing Vector
Vector is an open source host-level performance monitoring framework, which exposes hand-picked, high-resolution system and application metrics to every engineer’s browser. Having the right metrics available on demand and at a high resolution is key to understanding how a system behaves and correctly troubleshooting performance issues. Previously, we’d login to instances as needed, run a variety o
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Bending pause times to your will with Generational ZGC
Scalable Annotation Service — Marken