How Netflix microservices tackle dataset pub-sub
By Ammar Khaku IntroductionIn a microservice architecture such as Netflix’s, propagating datasets from a single source to multiple downstream destinations can be challenging. These datasets can represent anything from service configuration to the results of a batch job, are often needed in-memory to optimize access and must be updated as they change over time. One example displaying the need for d
OrpheusDB
With the increasing number of individuals performing data science, in every organization and team, there is a proliferation of dataset versions at various stages of data analysis. More often than not, these dataset versions are stored in an ad-hoc manner in shared file systems, leading to massive redundancy and duplication, and making it impossible to keep track of and find specific versions. Orph
Home Page | Pachyderm
import cv2 import numpy as np from matplotlib import pyplot as plt import os # edges.py reads an image and outputs transformed image def make_edges(image): img = cv2.imread(image) tail = os.path.split(image)[1] edges = cv2.Canny(img,100,200) plt.imsave(os.path.join("/pfs/out", os.path.splitext(tail)[0]+'.png?as=webp'), edges, cmap = 'gray') # walk images directory and call make_edges on every file
2015-06-qcon-microservices
Microservices and the art of taming the Dependency Hell Monster Michael Bryzek Cofounder & ex-CTO Gilt @mbryzek mbryzek@alum.mit.edu Dependency Hell • What is it and how does it happen? • How do we mitigate? • API design must be First Class • Backward and Forward Compatibility • Accurate Documentation • Generated client libraries http://en.wikipedia.org/wiki/Dependency_hell Dependency hell is a c
DataHub
DataHub A Data Ecosystem for Individuals, Teams and People What is DataHub? For End Users: A way to store your data centrally, without having to set up your own database A way to collaborate with others A way to seamlessly share your data with friends and colleagues A suite of tools to process your data For Developers: A database-agnostic, language-agnostic, MITREid (OpenID + OAuth2) integrated pl
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Are you still not using Version Control for Data?
Evolving HTTP APIs
API Evolution for REST/HTTP APIs
Dat Protocol Foundation
Why Git and SVN Fail at Managing Dataset Versions
Principles of Dataset Versioning: Exploring the Recreation/Storage Tradeoff
Introducing SchemaVer for semantic versioning of schemas