A Billion Taxi Rides on Amazon EMR running Presto
I have 15 years of consulting & hands-on build experience with clients in the UK, USA, Sweden, Ireland & Germany. Past clients include Bank of America Merrill Lynch, Blackberry, Bloomberg, British Telecom, Ford, Google, ITV, LeoVegas, News UK, Pizza Hut, Royal Mail, T-Mobile, Williams Formula 1, Wise & UBS. I hold both a Canadian and a British passport. My CV, Twitter & LinkedIn. This blog post wi
Scaling your analytics with Amazon EMR
The document discusses using Amazon EMR to scale analytics workloads on AWS. It provides an overview of EMR and how it allows users to easily run Hadoop clusters on AWS. It discusses how EMR allows tuning clusters and reducing costs by using Spot instances. It also discusses using various AWS services like S3, HDFS and integrating various Hadoop ecosystem tools on EMR. It provides examples of usin
AWS re:Invent 2015 | (BDT305) Amazon EMR Deep Dive and Best Practices
Amazon EMR is one of the largest Hadoop operators in the world. In this session, we introduce you to Amazon EMR design patterns such as using Amazon S3 instead of HDFS, taking advantage of both long and short-lived clusters, and other Amazon EMR architectural best practices. We talk about how to scale your cluster up or down dynamically and introduce you to ways you can fine-tune your cluster. We
Submitting User Applications with spark-submit | Amazon Web Services
AWS Big Data Blog Submitting User Applications with spark-submit Francisco Oliveira is a consultant with AWS Professional Services Customers starting their big data journey often ask for guidelines on how to submit user applications to Spark running on Amazon EMR. For example, customers ask for guidelines on how to size memory and compute resources available to their applications and the best reso
Data Engineering at VOYAGE GROUP #jawsdays
14. Targeting Infrastructure ELB EC2 EC2 EC2 EC2 request EC2 S3 DynamoDB EMR ELB EC2 EC2 EC2 EC2 http api fluentd fluentd (aggregator) out_exec_filter out_dynamodb servlet (scala) dynamic-dynamo EC2 Growth Forecast EC2 EC2 VPC 1 VPC 1 VPC 2 (targeting) VPC Peering VPC Peering ephemeral cluster 15. ELB EC2 EC2 EC2 EC2 request EC2 S3 DynamoDB EMR ELB EC2 EC2 EC2 EC2 http api fluentd fluentd (aggregator)
Indexing Common Crawl Metadata on Amazon EMR Using Cascading and Elasticsearch | Amazon Web Services
AWS Big Data Blog Indexing Common Crawl Metadata on Amazon EMR Using Cascading and Elasticsearch Hernan Vivani is a Big Data Support Engineer for Amazon Web Services A previous post showed you how to get started with Elasticsearch and Kibana on Amazon EMR. In that post, we installed Elasticsearch and Kibana on an Amazon EMR cluster using bootstrap actions. This post shows you how to build a simple
Amazon EMR Best Practices
Amazon Web Services – Best Practices for Amazon EMR August 2013 Page 1 of 38 Best Practices for Amazon EMR Parviz Deyhim August 2013 (Please consult http://aws.amazon.com/whitepapers/ for the latest version of this paper) Amazon Web Services – Best Practices for Amazon EMR August 2013 Page 2 of 38 Table of Contents Abstract..........................................................................
Apache HBase on Amazon EMR – Real-time Access to Your Big Data | Amazon Web Services
AWS News Blog Apache HBase on Amazon EMR – Real-time Access to Your Big Data All Your Base AWS has already given you a lot of storage and processing options to choose from, and today we are adding a really important one. You can now use Apache HBase to store and process extremely large amounts of data (think billions of rows and millions of columns per row) on AWS. HBase offers a number of powerfu
Lemur: Declarative launching of Hadoop jobs on Elastic-Mapreduce
In this post, I'm announcing a new tool for launching Hadoop jobs locally and on AWS's elastic-mapreduce, (EMR) which I call Lemur. I built this tool for The Climate Corporation (TCC), purveyors of Weather Insurance to the Agriculture Industry. I want to thank TCC for allowing me to open source this work, as well as a great deal of feedback and the original use cases. Overview Lemur is a tool to
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Migrate to Apache HBase on Amazon S3 on Amazon EMR: Guidelines and Best Practices | Amazon Web Services
Logをs3とredshiftに格納する仕組み
Building a Sustainable Data Platform on AWS
Getting Started With Scalding and Amazon Elastic MapReduce
GitHub - mlimotte/lemur: Lemur is a tool to launch hadoop jobs locally or on EMR, based on a configuration file, referred to as a jobdef. The jobdef file describes your EMR cluster, local environment, pre- and post-actions and zero or more "steps".