Elixir Flowでlazyな並列分散処理 - Qiita
Flowとは GenStageのWrapper. 普段コレクションを操作する時に使うEnumっぽいインターフェースなので親しみやすい. YoutubeにElixir作者のJoséの解説が上がっている. https://www.youtube.com/watch?v=srtMWzyqdp8&feature=youtu.be&t=244 GenStageとは コレクションをlazyかつ並列分散処理出来るElixirのbehaviour. AkkaのBack-Pressureなどに影響された. Back-Pressureとは 要求ドリブン(demand-driven)でデータの生成主体(producer), 消費主体(consumer)間で、cosumerの要求に従ってproducerがデータを渡すことでデータの流量を調節する仕組み。ConsumerはFlowを使うことでstageとして並列に並
The Token Approach for Architecting Flow in Elixir · rrrene
The Token Approach for Architecting Flow in Elixir 30 Apr 2018 // posted in Architecting Flow in Elixir In the last two posts we looked at the Token approach and how we can leverage metaprogramming to build pipelines and custom DSLs processing this Token. The Tokenis just a struct holding relevant data for the execution of your program: # This struct is handed down during the execution of your pro
Using Metaprogramming for Architecting Flow in Elixir · rrrene
Using Metaprogramming for Architecting Flow in Elixir 09 Apr 2018 // posted in Architecting Flow in Elixir TL;DR You can adapt the ideas presented in the last article to any kind of application. Macros can be used to build a Plug-like DSL for your specific use-case. But be careful: Use metaprogramming wisely and only where it’s a good fit. Do not try to build the “Plug for everything”. In the last
Advanced Techniques for Architecting Flow in Elixir · rrrene
Advanced Techniques for Architecting Flow in Elixir 26 Mar 2018 // posted in Architecting Flow in Elixir In the last post we explored how we can use either |> or with to model how data flows through our program. There is, however, a third concept to model flow in your applications: to hand down a “token” during the execution of your program. This token contains all the information necessary for yo
nerves.build
Photography Phoenix GenStage & Flow GeoCoding Personal Cloud When I first heard about GenStage and Flow my thought was to use it in a photo processing project that I had on the backburner. With so many photos to manage, and not just digital pictures, but scans, original art as well as older digital photos with sketchy data attached. I’ve never been happy with the organizing tools available in the
初心者のElixir |> Flow,GenStage |> Concurrent MapReduce - Qiita
When 2017/04/01 At Elixir Conf Japan 2017 LT Self Introduction ちきさん (Tomohiro Noguchi) Twitter/GitHub/Qiita: @ovrmrw I've started to learn Elixir 1 month ago. What I talk about Reading a JSON file and doing MapReduce. (JSONファイルを読み込んでMapReduceしたい。) Differences between Eager, Lazy and Flow(Concurrent). (Eager, Lazy, Flow(Concurrent)の違いについて。) Flow & GenStage is extremely fast. (Flowめちゃくちゃ速い。) === Git
Architecting Flow in Elixir Programs: An Introduction · rrrene
Architecting Flow in Elixir Programs: An Introduction 30 Jan 2018 // posted in Architecting Flow in Elixir Getting the “flow” of a program is one of the first things I do when looking at open source software or joining a new project at work. The easier it is to grasp how data flows through a program, the easier it is for me as a developer to estimate the impact of changes (and the business of soft
Quaff that potion: saving $millions with Elixir and Erlang - NextRoll
We slashed our DynamoDB costs by over 75% using Kinesis, DynamoDB streams, and Erlang/OTP (and now Elixir) to implement a global cache warming system. We present that system and two new open-source libraries for processing Kinesis and DynamoDB streams in a similar way using Elixir and Erlang. 15-20 minute read AdRoll uses Erlang/OTP as the basis for several internal products, including a real-time
Designing a P2P Lending platform with Elixir in mind
With this post, I want to share with you the design process on one of our latest projects — a P2P marketplace that was intended to be used by one of largest online lenders and all kinds of investors for securitization of their portfolios. And almost insane stack change that turned out to be a great idea. Why? To attract new engineers and backers to Elixir community. I do not consider myself an ent
How to use Elixir’s GenStage.Flow for image resizing
Image resizing is often a pretty important (but not often thought about) feature for many websites. We all want to make pages load fast and look good, and that means sending images at the size they’re going to be displayed, and hopefully optimised to minimise file size. At carwow, one of the big sources of images is stock photos of cars, which look something like this: We receive batches of these
GenStage and Flow - José Valim (Lambda Days 2017)
http://www.lambdadays.org/lambdadays2017/jose-valim José will explore the rationale and design decisions behind GenStage and Flow, two abstractions that have been researched and now implemented in the Elixir programming with a focus on back-pressure, concurrency and data processing.
Using Elixir's GenStage and Flow to build product recommendations - Binary Consulting
Background Flow allows developers to express computations on collections, similar to the Enum and Stream modules, although computations will be executed in parallel using multiple GenStages. The canonical example of Flow, taken from the GitHub repository, demonstrates how to count words in a document in parallel: File.stream!("path/to/some/file") |> Flow.from_enumerable() |> Flow.flat_map(&String.
Tuning Elixir GenStage/Flow pipeline processing
There are many use cases for GenStage/Flow. One of such cases that I recently faced was to get some records from PostgreSQL database, download files associated with these records from Amazon S3, extract text from these files and index this text into ElasticSearch. This task can be represented as a pipeline with four steps: SELECT record from database DOWNLOAD pdf file EXTRACT text from that file I
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
GitHub - plataformatec/flow: Computational parallel flows on top of GenStage
Designing Token APIs for Architecting Flow in Elixir · rrrene
How to partition by key and join back in order with Elixir Flow
Flow — Flow v1.2.4
Adding an Email Verification Flow With Phoenix - DockYard
send/receive messages in Elixir and rate control
Using Elixir's GenStage and Flow to build product recommendations