はじめに 前提 アメリカで働くためのビザ 業務経験 2023年のアメリカのテック業界の状況 具体的な就活のステップ ソフトウェアエンジニアのインタビューで求められることの抽象的な理解 レジュメ Job Descriptionから逆算してレジュメを作る 一枚におさめる 数字を用いてスケールとビジネスインパクトを示す なるべく隙間を埋める フォーマット添削ツールにかける レビューを受ける ネットワーキング・リファラル 応募する アメリカの就活はNumber Game 採用のトレンドを追う 時期を見計らう Linkedinで最新の求人を見つける方法 Promotedをすべて非表示にする "Most Recent"順にする 検索クエリを工夫する 設定をブックマークする 時間を決めて巡回する コーディングインタビュー対策 アルゴリズムの地図を脳内に作る 大学やCouseraでアルゴリズムの授業を取る
Amazon SQSでFIFOだからってシステム全体が Exactly-Once になると思ったら大間違いだっていう話 - Smoky God Express
TL; DR Amazon SQS で Exactly-Once なキューを使おうとも冪等な処理を書くべき キューが Exactly-Once であるという性質はシステム全体が Exactly-Once になることを保証できない 結局マルチデータソースへの書き込みの問題が残る Designing Data-Intensive Applications (邦訳: データ指向アプリケーションデザイン） が良書でした 邦訳は未読1ですが原著の内容がいいのできっとだいじょうぶでしょう Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems 作者: Martin Kleppmann出版社/メーカー: O'Reilly Media発売日: 2017/
Goとマルチコアスケール実装
マルチコア化の未来予測 半世紀前にSF映画「2001年宇宙の旅」に登場するコンピューターHAL-9000が並列コンピューティングの未来を示しました。マルチコアで構成されたコンピューターの物理コアを取り除いてもすぐにクラッシュせずに性能ダウンして処理が継続するという演出がありました。 当時ですらシングルコアコンピューティングの限界が予想されていて、現状のコンピューティングがマルチコア化しているという未来をしっかり予測できていたことがわかります。 演出はコア数に応じてコンピューティング性能がスケールしていることを表現しています。これはマルチコアスケールするソフトウェア実装の未来を示していたと思います。 シングルコア性能向上の頭打ち 2003年以降あたりはCPUの動作周波数が伸び悩み出したところ。 https://queue.acm.org/detail.cfm?id=2181798 より その
The Four Innovation Phases of Netflix’s Trillions Scale Real-time Data Infrastructure
My name is Zhenzhong Xu. I joined Netflix in 2015 as a founding engineer on the Real-time Data Infrastructure team and later led the Stream Processing Engines team. I developed an interest in real-time data in the early 2010s, and ever since believe there is much value yet to be uncovered. Netflix was a fantastic place to be surrounded by many amazing colleagues. I can’t be more proud of everyone
Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region
November, 25th 2020 We wanted to provide you with some additional information about the service disruption that occurred in the Northern Virginia (US-EAST-1) Region on November 25th, 2020. Amazon Kinesis enables real-time processing of streaming data. In addition to its direct use by customers, Kinesis is used by several other AWS services. These services also saw impact during the event. The trig
By Andrew Nguonly, Armando Magalhães, Obi-Ike Nwoke, Shervin Afshar, Sreyashi Das, Tongliang Liu, Wei Liu, Yucheng Zeng BackgroundOver the next few years, most content on Netflix will come from Netflix’s own Studio. From the moment a Netflix film or series is pitched and long before it becomes available on Netflix, it goes through many phases. This happens at an unprecedented scale and introduces
Wasmtime Reaches 1.0: Fast, Safe and Production Ready!
As of today, the Wasmtime WebAssembly runtime is now at 1.0! This means that all of us in the Bytecode Alliance agree that it is fully ready to use in production. In truth, we could have called Wasmtime production-ready more than a year ago. But we didn’t want to release just any WebAssembly engine. We wanted to have a super fast and super safe WebAssembly engine. We wanted to feel really confiden
Interesting Programming Languages an opinionated collection of programming languages. Created: Feb 19, 2020 by Pradeep Gowda Updated: May 17, 2024 Tagged: programming-language An opinionated collection of programming languages. This is a list of what I consider interesting. Requests to add to this list are welcome, but there is no guarantee I’ll include them. As you can see many mainstream program
Building Netflix’s Distributed Tracing Infrastructure
“@Netflixhelps Why doesn’t Tiger King play on my phone?” — a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue — which is difficult when troubleshooting distributed systems. Investigating a video streaming failure consists of inspecting all aspects of a member account. In our previous blog post we introduced Edgar, our t
TL;DRWarpStream is an Apache Kafka® protocol compatible data streaming platform built directly on top of S3. It's delivered as a single, stateless Go binary so there are no local disks to manage, no brokers to rebalance, and no ZooKeeper to operate. WarpStream is 5-10x cheaper than Kafka in the cloud because data streams directly to and from S3 instead of using inter-zone networking, which can be
One of the great parts of the hacker culture 1 and the GNU way is the invention 2 of pipes and the idea 3 that one program should do one thing and do it well. Each running program (process) has one input stream (called standard input or STDIN) and one output stream (called standard output or STDOUT) and also one additional output stream for logging/errors/warnings (STDERR). We can connect programs
This is an upcoming high performance computing book titled “Algorithms for Modern Hardware” by Sergey Slotin. Its intended audience is everyone from performance engineers and practical algorithm researchers to undergraduate computer science students who have just finished an advanced algorithms course and want to learn more practical ways to speed up a program than by going from $O(n \log n)$ to $
Introduction⌗ Postgres queue tech is a thing of beauty, but far from mainstream. Its relative obscurity is partially attributable to the cargo cult of “scale”. The scalability cult has decreed that there are several queue technologies with greater “scalability” than Postgres, and for that reason alone, Postgres isn’t suitably scalable for anyone’s queuing needs. The cult of scalability would rathe
On the Way to Democratized Stream Processing: RisingWave’s Roadmap - RisingWave: Open-Source Streaming Database
Two months ago, we open-sourced RisingWave, a cloud-native streaming database. RisingWave is developed on the mission to democratize stream processing — to make stream processing simple, affordable, and accessible. You may check out our recent blog, document, and source code for more information about RisingWave. Rome was not built in a day, and neither are database systems. We started developing
最長一致パターンに基づく高速・高精度な日本語形態素解析
ynaga@iis.u-tokyo.ac.jp 1/2 1/20 1,000,000 / C++ 1000 http://www.tkl.iis.u-tokyo.ac.jp/∼ynaga/jagger 1 Twitter Zoom, Slack [1] GPU [2, 3] [4, 5] ( ) () (MeCab, Vaporetto) MeCab 15 Vaporetto 10 (M2 MacBook Air 1,000,000 /) 2 [6] ( ) [7, 8] [9, 10] [11] ― 351 ― 言語処理学会 第29回年次大会 発表論文集 (2023年3月) This work is licensed by the author(s) under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Algor
Preview: AWS Proton – Automated Management for Container and Serverless Deployments | Amazon Web Services
AWS News Blog Preview: AWS Proton – Automated Management for Container and Serverless Deployments Today, we are excited to announce the public preview of AWS Proton, a new service that helps you automate and manage infrastructure provisioning and code deployments for serverless and container-based applications. Maintaining hundreds – or sometimes thousands – of microservices with constantly changi
AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sources
AWS Lambda now supports Parallelization Factor, a feature that allows you to process one shard of a Kinesis or DynamoDB data stream with more than one Lambda invocation simultaneously. This new feature allows you to build more agile stream processing applications on volatile data traffic. By default, Lambda invokes a function with one batch of data records from one shard at a time. For a single ev
Hyperdrive: making databases feel like they’re global
Hyperdrive: making databases feel like they’re global09/28/2023 This post is also available in 简体中文, 日本語, 한국어, Deutsch, Français and Español. Hyperdrive makes accessing your existing databases from Cloudflare Workers, wherever they are running, hyper fast. You connect Hyperdrive to your database, change one line of code to connect through Hyperdrive, and voilà: connections and queries get faster (
Single-table vs. multi-table design in Amazon DynamoDB | Amazon Web Services
AWS Database Blog Single-table vs. multi-table design in Amazon DynamoDB This is a guest post by Alex DeBrie, an AWS Hero. For people learning about Amazon DynamoDB, the idea of single-table design is one of the most mind-bending concepts out there. Rather than the relational notion of having a table per entity, DynamoDB tables often include multiple different entities in a single table. You can r
What we look for in a resume
I read every single one of the resumes we receive. Sometimes, I’d talk to a candidate and see that what we perceived as their strongest aspects actually weren’t included in their resume. Occasionally, a candidate would tell me that they didn’t expect their resume to still be screened by humans – had they known, they would have written their resume differently. The resume evaluation process is pret
Presto | Paper
Presto: SQL on Everything IEEE International Conference on Data Engineering (ICDE) Raghav Sethi, Martin Traverso, Dain Sundstrom, David Phillips, Wenlei Xie, Yutian Sun, Nezih Yigitbasi, Haozhun Jin, Eric Hwang, Nileema Shingte, Christopher Berner Abstract Presto is an open source distributed query engine that supports much of the SQL analytics workload at Facebook. Presto is designed to be adapti
Co-authors: Xiang Zhang and Jingyu Zhu Introduction The Lambda architecture has become a popular architectural style that promises both speed and accuracy in data processing by using a hybrid approach of both batch processing and stream processing methods. But it also has some drawbacks, such as complexity and additional development/operational overheads. One of our features for Premium members on
AWS Lambda Supports Failure-Handling Features for Kinesis and DynamoDB Event Sources
AWS Lambda now supports four failure-handling features for processing Kinesis and DynamoDB streams: Bisect on Function Error, Maximum Record Age, Maximum Retry Attempts, and Destination on Failure. These new features allow you to customize responses to data processing failures and build more resilient stream processing applications. Lambda processes data records from Kinesis and DynamoDB streams i
Amazon Rekognition Streaming Video Eventsでリアルタイムに人検知を行う - Taste of Tech Topics
この記事は AI/ML on AWS Advent Calendar 2022 12/23、および、アクロクエスト アドベントカレンダー 12/23 の記事です。 qiita.com こんにちは、Acroquest データサイエンスチーム YAMALEX メンバーの駿です。 早いもので2022年もあと一週間と少しになってしまいました。 皆さんは年を越す準備は万端でしょうか？ 私が住む社員寮では慌てて年越しそばを買ったり、餅を買ったり、ぎりぎりになってバタバタしています。 さて、今回は 2022年４月に発表された Amazon Rekognition の Streaming Video Events を使って、リアルタイムの人検知を試してみました。 エッジデバイスには Raspberry Pi を使いました。 検出結果の例 1. はじめに (1) Amazon Rekognition とは
Delta: A Data Synchronization and Enrichment Platform
Andreas Andreakis, Falguni Jhaveri, Ioannis Papapanagiotou, Mark Cho, Poorna Reddy, Tongliang Liu OverviewIt is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), providing advanced search capabilities (ElasticSearch etc.), caching (Memcached etc.), and more. Typically when
World Wide Web Consortium (W3C) brings a new language to the Web as WebAssembly becomes a W3C Recommendation
Following HTML, CSS and JavaScript, WebAssembly becomes the fourth language for the Web which allows code to run in the browser Read testimonials from W3C Members https://www.w3.org/ — 5 December 2019 — The World Wide Web Consortium (W3C) announced today that the WebAssembly Core Specification is now an official web standard, launching a powerful new language for the Web. WebAssembly is a safe, po
What Every Software Engineer Should Know about Apache Kafka: Events, Streams, Tables, Storage, Processing, And More
To help fellow engineers wrap their head around Apache Kafka and event streaming, I wrote a 4-part series on the Confluent blog on Kafka’s core fundamentals. In the series, we explore Kafka’s storage and processing layers and how they interrelate, featuring Kafka Streams and ksqlDB. In the first part, I begin with an overview of events, streams, tables, and the stream-table duality to set the stag
ksqlDB The database purpose-built for stream processing applications. Real-time Build applications that respond immediately to events. Craft materialized views over streams. Receive real-time push updates, or pull current state on demand. Kafka-native Seamlessly leverage your existing Apache Kafka® infrastructure to deploy stream-processing workloads and bring powerful new capabilities to your app
10 Things Serverless Architects Should Know | Amazon Web Services
AWS Architecture Blog 10 Things Serverless Architects Should Know Building on the first three parts of the AWS Lambda scaling and best practices series where you learned how to design serverless apps for massive scale, AWS Lambda’s different invocation models, and best practices for developing with AWS Lambda, we now invite you to take your serverless knowledge to the next level by reviewing the f
[レポート] The Modern Data Stack: Past, Present, and Future #futuredataconf | DevelopersIO
[レポート] The Modern Data Stack: Past, Present, and Future #futuredataconf 奈良県でリモートワーク中の玉井です。 9月8日〜9月9日の2日間、FUTURE DATA CONFERENCE 2020というオンラインイベントが開催されていました。今回、そのイベントの下記のウェビナーを受講したので、レポートします。 イベント全体の概要ですが、名前の通り、「データ分析（とそれに関するテクノロジー）の今後」について、多種多様な業界の方々が語るって感じのイベントのようです。 今回はその中の「The Modern Data Stack: Past, Present, and Future」というセッションについてレポートします。 ウェビナー情報 公式情報 ※本カンファレンスは、既に2021年分が開催済であり、ウェブサイトの内容も2021
![[レポート] The Modern Data Stack: Past, Present, and Future #futuredataconf | DevelopersIO](https://cdn-ak-scissors.b.st-hatena.com/image/square/fb430de839b0b06456078ad39cdefc962bdfc216/height=288;version=1;width=512/https%3A%2F%2Fdevio2023-media.developers.io%2Fwp-content%2Fuploads%2F2020%2F09%2FFuture_Data_Conference_2020.png)
ML Platform Meetup: Infra for Contextual Bandits and Reinforcement Learning
Infrastructure for Contextual Bandits and Reinforcement Learning — theme of the ML Platform meetup hosted at Netflix, Los Gatos on Sep 12, 2019. Contextual and Multi-armed Bandits enable faster and adaptive alternatives to traditional A/B Testing. They enable rapid learning and better decision-making for product rollouts. Broadly speaking, these approaches can be seen as a stepping stone to full-o
Big Data: Google Replaces YARN with Kubernetes to Schedule Apache Spark
Will real-time data processing replace batch processing? At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. Infrastructure teams at Meta leverage UPM to build SQL linters, catch user mistakes in SQL code, and perform data lineage analysis at scale. Executing SQL queries against our data warehouse is important to
※この投稿は米国時間 2020 年 8 月 21 日に、Google Cloud blog に投稿されたものの抄訳です。 編集者注: 本記事は Dataflow の開発に至った Google 内部の歴史と、Google Cloud サービスとしての Dataflow の機能、市場における他社製品との比較対照について掘り下げる 3 回シリーズのブログの第 1 回です。 Google のスマート分析プラットフォームの一部である Google Cloud の Dataflow は、ストリーム データとバッチデータの処理を統合するストリーミング分析サービスです。Dataflow に対する理解を深めるために、MillWheel から始まるその歴史も理解しておくとよいでしょう。 Dataflow の歴史Google の多くのプロジェクトと同様、MillWheel は 2008 年に小さなチームが考案し
Advent Calendar 2020 の 4 日目の記事です。 こんにちは、 wind-up-bird です。 前回に引き続き、ECS移行について書いていきたいと思います。 前編: 移行前の構成や課題、移行方針を記載しています。 後編: 移行後の構成や旧環境との変更点を記載しています。 ※前編をまだ読んでない人は是非チェックしてみてね！ 目次 目次 新環境 全体構成 詳細 AWS のリソース管理 デプロイ ロギング 移行作業 振り返り Build 時間の短縮 EC2とOSの管理から開放された 起動時間と費用 ハマった点など 設定ファイルの管理 ECS 標準デプロイアクション 16kB のログチャンクサイズ 最後に お決まり 新環境 この章では、ECS移行後の構成やデプロイ方法、移行方法を紹介していきたいと思います。 全体構成 全体的な構成は以下のようなイメージです。 1 つの ECS
![トレタのバックエンドを ECS へ移行した話 [後編] - トレタ開発者ブログ](https://cdn-ak-scissors.b.st-hatena.com/image/square/4b4633b0bb10df551f8ce485e8e2cb1184ab4f59/height=288;version=1;width=512/https%3A%2F%2Fcdn-ak.f.st-hatena.com%2Fimages%2Ffotolife%2Fw%2Fw1nd-up-b1rd%2F20201201%2F20201201153302.png)
Cover Photo by Alexander Shatov on Unsplash Netflix accounts for about 15% of the world's internet bandwidth traffic. Serving over 6 billion hours of content per month, globally, to nearly every country in the world. Building a robust, highly scalable, reliable, and efficient backend system is no small engineering feat but the ambitious team at Netflix has proven that problems exist to be solved.
In a previous post, I showed how Kafka can be used as the persistent storage for an embedded key-value store, called KCache. Once you have a key-value store, it can be used as the basis for other models such as documents, graphs, and even SQL. For example, CockroachDB is a SQL layer built on top of the RocksDB key-value store and YugaByteDB is both a document and SQL layer built on top of RocksDB.
How to build large-scale end-to-end encrypted group video calls
How to build large-scale end-to-end encrypted group video calls peter-signal on 15 Dec 2021 Signal released end-to-end encrypted group calls a year ago, and since then we’ve scaled from support for 5 participants all the way to 40. There is no off the shelf software that would allow us to support calls of that size while ensuring that all communication is end-to-end encrypted, so we built our own
Develop and test AWS Glue version 3.0 and 4.0 jobs locally using a Docker container | Amazon Web Services
AWS Big Data Blog Develop and test AWS Glue version 3.0 and 4.0 jobs locally using a Docker container Apr 2023: This post was reviewed and updated with enhanced support for Glue 4.0 Streaming jobs. Jan 2023: This post was reviewed and updated with enhanced support for Glue 3.0 Streaming jobs, ARM64, and Glue 4.0. AWS Glue is a fully managed serverless service that allows you to process data coming
From Static to Real-time: Introducing Incremental Builds in Gatsby Cloud Today I’m thrilled to announce the release of Incremental Builds on Gatsby Cloud. In January we announced Gatsby Builds, bringing you up to 60x faster builds for Gatsby sites compared to other solutions. Now, when you make a data change in a CMS, the Gatsby Cloud Incremental Builds feature will rebuild only what’s necessary—g
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
アメリカでソフトウェアエンジニアの職を探した - pco2699’s blog
Data Movement in Netflix Studio via Data Mesh
Interesting Programming Languages
Kafka is dead, long live Kafka
Relational pipes
Algorithms for Modern Hardware - Algorithmica
Choose Postgres queue technology
From Lambda to Lambda-less: Lessons learned
ksqlDB: The database purpose-built for stream processing applications.
Enabling static analysis of SQL queries at Meta
Dataflow の仕組み: 誕生秘話 | Google Cloud 公式ブログ
トレタのバックエンドを ECS へ移行した話 [後編] - トレタ開発者ブログ
Netflix System Design- Backend Architecture
Building A Relational Database Using Kafka
What is Gatsby Incremental Builds? | Gatsby