Applied machine learning at facebook a datacenter infrastructure perspective HPCA18
This document discusses recommendations and machine learning at Netflix. It provides an overview of: - How Netflix provides personalized recommendations on member homepages to help them find content to watch. - Netflix's experimentation cycle of designing experiments, collecting data, generating features, training models, and doing A/B testing. - How Netflix handles "facts" or input data for recom
facebookの13億ユーザーを支えるロードバランサーの話 - stanaka's blog
最近、SREが話題ですね。 tech.mercari.com www.wantedly.com ということでSREについて調べてたら、SREconなんてものが開催されていたので中を見てたら、「Building a Billion User Load Balancer」というタイトルでFacebookのDNS〜LBまでの話があったので、そのメモです。 Building a Billion User Load Balancer | USENIX tl;dr tinydns + IPVS で Facebook規模はいける httpsの接続確立はかなり重い(RTTの4倍 = RTT 150msとするとGETまで600ms)ので、太平洋越えとかは厳しい httpsを終端させるCDNとかは活用の可能性ありそう (国内だけを考慮するなら影響は軽微かも) メモ L4 LB shiv (IPVS + pyt
Web performance: Cache efficiency exercise
Speed is a consideration for any website, whether it's for the local barbershop or Wikipedia, with its huge repository of knowledge. It's a feature that shouldn't be ignored. This is why caching is important — a great way to make websites faster is to save parts of them so they don't have to be calculated or downloaded again on the next visit. My team was recently having a discussion about the par
Introducing osquery
Maintaining real-time insight into the current state of your infrastructure is important. At Facebook, we’ve been working on a framework called osquery which attempts to approach the concept of low-level operating system monitoring a little differently. Osquery exposes an operating system as a high-performance relational database. This design allows you to write SQL-based queries efficiently and e
Facebook's software architecture
On distributed systems broadly defined and other curiosities. The opinions on this site are my own. I had summarized/discussed a couple papers (Haystack, Memcache caching) about Facebook's architecture before. Facebook uses simple architecture that gets things done. Papers from Facebook are refreshingly simple, and I like reading these papers. Two more Facebook papers appeared recently, and I brie
Scaling Memcache at Facebook | USENIX
by Michael Piatek Responsiveness is essential for web services. Speed drives user engagement, which drives revenue. To reduce response latency, modern web services are architected to serve as much as possible from in-memory caches. The structure is familiar: a database is split among servers with caches for scaling reads. Over time, caches tends to accumulate more responsibility in the storage sta
Instagram moves 20 billion images to Facebook servers - SiliconANGLE
Instagram announced last week that it’s picked up its billions of images stored in Amazon Web Services (AWS) and dumped them into Facebook’s own servers in one of the largest data migration operations ever undertaken. News of the move came from this interview with Facebook infrastructure engineer and Open Compute Foundation program developer Charlie Manese. Manese revealed that the massive migrati
Scaling the Facebook data warehouse to 300 PB
At Facebook, we have unique storage scalability challenges when it comes to our data warehouse. Our warehouse stores upwards of 300 PB of Hive data, with an incoming daily rate of about 600 TB. In the last year, the warehouse has seen a 3x growth in the amount of data stored. Given this growth trajectory, storage efficiency is and will continue to be a focus for our warehouse infrastructure. There
An Analysis of Facebook Photo Caching
An Analysis of Facebook Photo Caching Qi Huang∗, Ken Birman∗, Robbert van Renesse∗, Wyatt Lloyd†‡, Sanjeev Kumar‡, Harry C. Li‡ ∗Cornell University, †Princeton University, ‡Facebook Inc. Abstract This paper examines the workload of Facebook’s photo- serving stack and the effectiveness of the many layers of caching it employs. Facebook’s image-management infrastructure is complex and geographically
An analysis of Facebook photo caching
Every day people upload more than 350 million photos to Facebook (as of Dec 2012) and view many more in their News Feeds and on their friends’ Timelines. Facebook stores these photos on Haystack machines that are optimized to store photos. But there is also a deep and distributed photo-serving stack with many layers of caches that delivers photos to people so they can view them. We recently publis
Facebookの継続的デプロイメント - ワザノバ | wazanova
https://www.facebook.com/publications/514128035341603/ 1日500件、3,000ファイルに及ぶ本番アップ フロントエンドのコードは1050万行、内850万行がPHP 開発エンジニア1,000名とリリースエンジニア3名 QAやテスターは存在しない 自分でプロジェクトを選ぶ & 自己責任のカルチャーが強い。 1/3のファイルが一人のエンジニア、1/4が二人のエンジニアでメンテされている。 フロントエンドの本番コードベースは一つのものを共有 日常業務ではローカルのgitを利用。本番アップ可能になれば、中央のレポジトリにマージして、それからSubversion（過去の経緯で使っている。）にコミットする 同じエンジニアがコードをコミットする間隔は中央値で10時間 本番にプッシュする前に、担当エンジニア自身でのユニットテストを終え、同僚によるコード
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
The Evolution of ML Platforms at Facebook Recap
GitHub - facebook/sapling: A Scalable, User-Friendly Source Control System.
Zero Downtime Release: Disruption-free Load Balancing of a Multi-Billion User Website
OCPREG19 - Facebook Challenges and Opportunities of Architecting AI Systems at Facebook Datacenter
【ECCV 2018】Exploring the Limits of Weakly Supervised Pretraining
GitHub - facebook/wangle: Wangle is a framework providing a set of common client/server abstractions for building services in a consistent, modular, and composable way.
GitHub - facebook/mcrouter: Mcrouter is a memcached protocol router for scaling memcached deployments.
Scaling Userspace @ Facebook
F4 - Photo Storage at Facebook
