GitHub - rmax/scrapy-redis: Redis-based components for Scrapy.Distributed crawling/scraping You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Distributed post-processing Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scrapy plug-and-play components Scheduler + Duplication Filter, Item Pipeline, Bas
scrapy_tdd/test/test_helpers.py at master · rrschmidt/scrapy_tdd
