NetflixOSS: Announcing Hollow
“If you can cache everything in a very efficient way, you can often change the game” We software engineers often face problems that require the dissemination of a dataset which doesn’t fit the label “big data”. Examples of this type of problem include: Product metadata on an ecommerce siteDocument metadata in a search engineMetadata about movies and TV shows on an Internet television networkWhen f
Vizceral Open Source
Previously we wrote about our traffic intuition tool, Flux. We have some announcements and updates to share about this project. First, we have renamed the project to Vizceral. More importantly, Vizceral is now open source! Open SourceVizceral transformed the way we understand and digest information about the state of traffic flowing into the Netflix control plane. We wanted to be able to intuit de
Meson: Workflow Orchestration for Netflix Recommendations
At Netflix, our goal is to predict what you want to watch before you watch it. To do this, we run a large number of machine learning (ML) workflows every day. In order to support the creation of these workflows and make efficient use of resources, we created Meson. Meson is a general purpose workflow orchestration and scheduling framework that we built to manage ML pipelines that execute workloads
Kafka Inside Keystone Pipeline
In summary, the Keystone pipeline is a unified event publishing, collection, and routing infrastructure for both batch and stream processing. We have two sets of Kafka clusters in Keystone pipeline: Fronting Kafka and Consumer Kafka. Fronting Kafka clusters are responsible for getting the messages from the producers which are virtually every application instance in Netflix. Their roles are data co
The Netflix Way to deal with Big Data Problems
The Netflix Way to deal with Big Data Problems 1. The way to deal with Big data problems Monal Daxini March 2016 2. Monal Daxini Real Time Data Infrastructure Senior Software Engineer, Netflix https://www.linkedin.com/in/monaldaxini @monaldax #Netflix #Keystone 3. We help Produce, Store, Process, Move Events @ scale 4. Tell me more... ● Big Data Ecosystem @ Netflix ● How we built a scalable event
Stream-processing with Mantis
Back in January of 2014 we wrote about the need for better visibility into our complex operational environments: The core of the message in that post was about the need for fine-grained, contextual and scalable insights into the experiences of our customers and behaviors of our services. While our execution has evolved somewhat differently from our original vision, the underlying principles behind
Evolution of the Netflix Data Pipeline
Our new Keystone data pipeline went live in December of 2015. In this article, we talk about the evolution of Netflix’s data pipeline over the years. This is the first of a series of articles about the new Keystone data pipeline. Netflix is a data-driven company. Many business and product decisions are based on insights derived from data analysis. The charter of the data pipeline is to collect, ag
Netflix and Containers - Titus
Slides from the 2016/01/28 Advanced Amazon Web Services (AWS) Meetup. Netflix overviewed the usage of containers at Netflix. We covered technologies we are working on in the runtime (Titus) and developer experience (Newt). We talked about how the Titus container management system was different from others and our journey with Docker, Mesos, Netflix Fenzo and eventually Amazon Elastic Container Ser
Spring Cloud Netflixを使おう #jsug
Pivotal認定講師によるSpring Framework 5.1 & Spring Boot 2.1ハンズオン！ #jjug_cccMasatoshi Tada
Spring Cloud Netflixを読んだ
Springが、Netflixが公開している彼らのOSS群を説明している記事を読みました。ちょっと、記憶に止めておきたい内容を探っているのでそのメモがてら。 記事: http://cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.html これは、Netflixのmicroservicesを構成するツール群を説明していました。 Service Discovery Eureka Clients Circuit Breaker Hystrix Intelligent Routing Zuul Client Side Load Balancing Ribbon Service Discovery: Eureka Clients Service Discoveryはmicsoservicesベースのアーキテクチャとして大事な要素 C
NetflixのgithubリポジトリはWeb技術の百貨店だった - DiaryException
検索しているとなにかとNetflixのgithubリポジトリがヒットするので、全部（2015/07/18現在分）調査してみた。 github APIで https://github.com/Netflix のリストを全部取得して、名前・概要・URL・最終更新日時 (なんの更新だ?) を抽出。 AWS用のプロダクトが多かったのでまずそれらと、その他という分類にした。その他はほとんどがJavaライブラリ・システムだが、一部WebアプリケーションやPythonライブラリがある。 日本語での説明はReadmeやWikiを見て書いているが、理解が正しくないかもしれない。 AWS用 aws-autoscaling Tools and Documentation about using Auto Scaling URL: https://github.com/Netflix/aws-autoscalin
NetflixのHystrixにも使われるCircuit Breaker patternを調べてみた
Testing Strategies In A Microservice Architectureを読んだを読んでいる途中に出てきた、Circuit Breakerと呼ばれる機構を調べてみました。Martin Fowler氏がこの記事で言及しているものでした。 このCircuit Breaker patternは、Release It! 本番用ソフトウェア製品の設計とデプロイのためにで描かれているような、本番環境化において発生する、複数システムが関係するからこそ発生する障害を抑えることも目的としたデザインパターンのようです。「複数システムが関係するからこそ発生する障害」とは、一部システムの負荷が高まりタイムアウトするといったことを含みます。 内容自体は、 障害検出のための共有のオブジェクト(Circuit Breacker)を用意して、監視・検出できるようにする ということらしいです。Ne
NETFLIXのSpinnakerをさわってみた備忘禄 - ゆるふわカウンターアタック
マルチクラウドのインフラアプリデリバリの自動化と管理ツールがNETFLIXから11/17にオープンソースで公開されたのでさわって見た。 NETFLIXがGoogle,Microsoft,Pivotalと共同開発したそうです。 名前は「Spinnaker」(スピネーカー)、意味は"三角形の帆"ですが由来は不明です。 NETFLIXブログ Netflix、マルチクラウド対応の継続的デリバリを実現する「Spinnaker」をオープンソースで公開 － Publickey Pivotalブログ Cloud Native Continuous Delivery With Netflix Spinnaker | Pivotal P.O.V. 本家 The Netflix Tech Blog: Global Continuous Delivery with Spinnaker Spinnaker: Glo
From Chaos to Control
by Leena Janardanan, Bruce Wobbe, Vilas Veeraraghavan IntroductionMerchandising Application Platform (MAP) was conceived as a middle-tier service that would handle real time requests for content discovery. MAP does this by aggregating data from disparate data sources and implementing common business logic into one distinct layer. This centralized layer helps provide common experiences across devic
Fenzo: OSS Scheduler for Apache Mesos Frameworks
Bringing Netflix to our millions of subscribers is no easy task. The product comprises dozens of services in our distributed environment, each of which is operating a critical component to the experience while constantly evolving with new functionality. Optimizing the launch of these services is essential for both the stability of the customer experience as well as overall performance and costs. T
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Netflix Keystone—Cloud scale event processing pipeline
GitHub - Netflix/ribbon: Ribbon is a Inter Process Communication (remote procedure calls) library with built in software load balancers. The primary usage model involves REST calls with various serialization scheme support.
Application Resilience Engineering & Operations at Netflix
GitHub - Netflix/archaius: Library for configuration management API
GitHub - Netflix/Fenzo: Extensible Scheduler for Mesos Frameworks
