Yahoo! GroupsHello folks, We are in the starting phase of a project, and we are currently wondering whether Heritrix or Nutch is the best choice of crawler for us. Our project: Basically, we're going to set up Hadoop and crawl the web for images. We will then run our own indexing software on the images stored in HDFS based on the Map/Reduce facility in Hadoop. We will not use other indexing than our own. Some
Web Spam Detection for Heritrix - Heritrix用Webスパム発見器 | 関口宏司のLuceneブログ
