from_settingsだけ定義すると、Item Pipelineごして有効にならない。 下記のように、initでちゃんと定義し直す必要がある。 class MySQLStorePipeline(object): @classmethod def from_settings(cls, settings): return cls(settings.get('DB_SETTING')) def __init__(self, db_settings): db.init_session(db_settings) def process_item(self, item, spider): shop = Shop( brand_id=1, name=item['name'], address=item['address']) db.db_session.add(shop) db.db_session
Airflow - "Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues
Awesome Streaming A curated list of awesome streaming (stream processing) frameworks, applications, readings and other resources. Inspired by other awesome projects. Website https://manuzhang.github.io/awesome-streaming/ is a more dynamic website where you can find updates of the awesome projects here. Table of Contents Streaming Engine Streaming Library Streaming Application IoT DSL Data Pipeline
Please add new entries at the bottom. In addition to this list, actively developed free/open-source systems should be registered at https://workflows.community/systems See also: https://github.com/pditommaso/awesome-pipeline Arvados - CWL-based distributed computing platform for data analysis on massive data sets. https://arvados.org/ https://github.com/arvados/arvados Apache Taverna http://www.ta
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
処理を実行中です
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く