Spidr - AboutSpidr is a versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use. Features Custom User-Agent strings.Custom proxy settings. Follows: a tags.iframe tags.frame tags.HTTP 300, 301, 302, 303 and 307 Redirects. Black-list or white-list URLs based upon: URL schemeHost namePort numberFull linkURL extension
