Big Data Governance: Hive Metastore Listener for Apache Atlas Use Cases
Big Data Governance: Hive Metastore Listener for Apache Atlas Use Cases At eBay, we are obsessed with data quality and governance. Because eBay's Hadoop platform hosts 500 PB of data running over 15,000 nodes, the focus on governance is of utmost importance. This article discusses our experiences handling data governance at scale. Data governance helps ensure that high data quality exists througho
GZinga: Seekable and Splittable Gzip
Generally, data compression techniques are used to conserve space and network bandwidth. Widely used compression techniques include Gzip, bzip2, lzop, and 7-Zip. According to performance benchmarks, lzop is one of the fastest compression algorithms, while bzip2 has a high compression ratio but is very slow. Gzip offers the lowest level of compression. Gzip is based on the DEFLATE algorithm, which
Statistical Anomaly Detection
Complex systems can fail in many ways and I find it useful to divide failures into two classes. The first consists of failures that can be anticipated (e.g. disk failures), will happen again, and can be directly checked for. The second class is failures that are unexpected. This post is about that second class. The tool for unexpected failures is statistics, hence the name for this post. A statis
Announcing Pulsar: Real-time Analytics at Scale
We are happy to announce Pulsar – an open-source, real-time analytics platform and stream processing framework. Pulsar can be used to collect and process user and business events in real time, providing key insights and enabling systems to react to user activities within seconds. In addition to real-time sessionization and multi-dimensional metrics aggregation over time windows, Pulsar uses a SQL-
Announcing Kylin: Extreme OLAP Engine for Big Data
We are very excited to announce that eBay has released to the open-source community our distributed analytics engine: Kylin (http://kylin.io). Designed to accelerate analytics on Hadoop and allow the use of SQL-compatible tools, Kylin provides a SQL interface and multi-dimensional analysis (OLAP) on Hadoop to support extremely large datasets. Kylin is currently used in production by various busine
Delivering eBay's CI Solution with Apache Mesos - Part I
Problem statement In eBay’s existing CI model, each developer gets a personal CI/Jenkins Master instance. This Jenkins instance runs within a dedicated VM, and over time the result has been VM sprawl and poor resource utilization. We started looking at solutions to maximize our resource utilization and reduce the VM footprint while still preserving the individual CI instance model. After much deli
Delivering eBay's CI Solution with Apache Mesos - Part II
In part I of this post we laid out in detail how to run a large Jenkins CI farm in Mesos. In this post we explore running the builds inside Docker containers and more: Overview Jenkins follows the master-slave model and is capable of launching tasks as remote Java processes on Mesos slave machines. Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed
REST Commander: Scalable Web Server Management and Monitoring
REST Commander: Scalable Web Server Management and Monitoring In the era of cloud and XaaS (everything as a service), REST/SOAP-based web services have become ubiquitous within eBay’s platform. We dynamically monitor and manage a large and rapidly growing number of web servers deployed on our infrastructure and systems. However, existing tools present major challenges when making REST/SOAP calls w
Using Spark to Ignite Data Analytics
At eBay we want our customers to have the best experience possible. We use data analytics to improve user experiences, provide relevant offers, optimize performance, and create many, many other kinds of value. One way eBay supports this value creation is by utilizing data processing frameworks that enable, accelerate, or simplify data analytics. One such framework is Apache Spark. This post descri
Cassandra Data Modeling Best Practices, Part 1
This is the first in a series of posts on Cassandra data modeling, implementation, operations, and related practices that guide our Cassandra utilization at eBay. Some of these best practices we’ve learned from public forums, many are new to us, and a few still are arguable and could benefit from further experience. September 2014 Update: Readers should note that this article describes data modeli
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Interactive Visual Search
Explainable Reasoning over Knowledge Graphs for Recommendation
Beyond Logos and Patterns: How We’re Training eBay’s AI to Understand Brands
ModaNet: A Large-scale Street Fashion Dataset with Polygon Annotations
Providing Metadata Discovery on Large-Volume Data Sets
Seven Tips for Visual Search at Scale
Large-Scale Product Image Recognition with Cloud TPUs
Publications | Research
Product Images for Online Shopping