GitHub - rmax/scrapy-redis: Redis-based components for Scrapy.
Distributed crawling/scraping You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Distributed post-processing Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scrapy plug-and-play components Scheduler + Duplication Filter, Item Pipeline, Bas
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
処理を実行中です
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
GitHub - istresearch/scrapy-cluster: This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
