データドリブン企業における、hadoop基盤とETL #hcj2016
Hadoop Conference Japan 2016 で発表した資料です。 http://www.eventbrite.com/e/hadoop-spark-conference-japan-2016-tickets-20809016328 この資料は前半部分です。ドワンゴ志村様による後半部分はこちら: http://www.slideshare.net/smrmkt/hadoopetl-niconico ---- (アブストラクト) ETL(Extract, Transform, Load) は、外部のソースからデータを抽出し、業務上必要な形に加工し、データ基盤にロードするための一連の技術です。Hadoopの登場により、スケーラブルかつ柔軟なスキーマ構造を持つデータレイヤーであるHDFSに代表される、ETLにおける多くの物理アーキテクチャに変化がもたらされました。しかし一方で、ETL
Workflow Engines for Hadoop
Building a reliable pipeline of data ingress, batch computation, and data egress with Hadoop can be a major challenge. Most folks start out with cron to manage workflows, but soon discover that doesn't scale past a handful of jobs. There are a number of open-source workflow engines with support for Hadoop, including Azkaban (from LinkedIn), Luigi (from Spotify), and Apache Oozie. Having deployed a
Genie - Hadoop Platform as a Service at Netflix
In a prior tech blog (http://nflx.it/XoySYR), we had discussed the architecture of our petabyte-scale data warehouse in the cloud. Salient features of our architecture include the use of Amazon’s Simple Storage Service (S3) as our "source of truth", leveraging the elasticity of the cloud to run multiple dynamically resizable Hadoop clusters to support various workloads, and our horizontally scalab
SQL on Hadoop in Taiwan
This document discusses SQL engines for Hadoop, including Hive, Presto, and Impala. Hive is best for batch jobs due to its stability. Presto provides interactive queries across data sources and is easier to manage than Hive with Tez. Presto's distributed architecture allows queries to run in parallel across nodes. It supports pluggable connectors to access different data stores and has language bi
Powering Interactive Data Analysis at Pinterest by Amazon Redshift
Amazon Redshift is a fast, petabyte-scale data warehouse that makes it easy to analyze your data for a fraction of the cost of traditional data warehouses. In this webinar, you will learn how to easily migrate your data from other data warehouses into Amazon Redshift, efficiently load your data with Amazon Redshift's massively parallel processing (MPP) capabilities, and automate data loading with
Hadoop 2 @Twitter, Elephant Scale. Presented at
1) Twitter migrated their Hadoop infrastructure from Hadoop 1 to Hadoop 2 to address scalability challenges and improve efficiency from tens of thousands of servers and hundreds of petabytes of data. 2) The migration involved extensive testing of Hadoop 2 configurations and dependent components before rolling out to production clusters supporting various use cases such as analytics, personalizatio
[AWSマイスターシリーズ] Amazon Elastic MapReduce (EMR)
1. © 2012 Amazon.com, Inc. and its affiliates. Allrights reserved. May not be copied, modified or distributed in w hole or in part w ithout the express consent of Amazon.com, Inc. AWS マイスターシリーズ Amazon Elastic MapReduce 2013.09.25 アマゾンデータサービスジャパン株式会社 ソリューションアーキテクト 大谷 晋平 2. © 2012 Amazon.com, Inc. and its affiliates. Allrights reserved. May not be copied, modified or distributed in w hole or in part
![[AWSマイスターシリーズ] Amazon Elastic MapReduce (EMR)](https://cdn-ak-scissors.b.st-hatena.com/image/square/5e37ea20134c298b161bece95a38cc60784bcf78/height=288;version=1;width=512/https%3A%2F%2Fcdn.slidesharecdn.com%2Fss_thumbnails%2F20130925aws-meister-regenerate-emrpublic-130926030316-phpapp01-thumbnail.jpg%3Fwidth%3D640%26height%3D640%26fit%3Dbounds)
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Hadoop World 2011: Hadoop Troubleshooting 101 - Kate Ting - Cloudera
SQL on Hadoop 比較検証 【2014月11日における検証レポート】
Python + Hive on AWS EMR で貧者のログサマリ
「Application Performance2010」でご紹介したHadoop講演資料を公開します