Answer (1 of 16): It always depends on the projects size and on the difficulty of the data sources. Crawling different data sources goes beyond simple crawler development or configuration. There are several issues a crawling project might be facing as: * Data discovery - need to crawl a websit...