RubyGems.org

cobweb

1.0.20

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

installgem install cobweb
Authors

Stewart McKee

82,114 total downloads 171 for this version
Owners

8cdb978dffae6ab207dbdeb60d01af2f

Licenses

MIT

Gemfile
gem 'cobweb', '~> 1.0.20'
Versions
  1. 1.0.20 August 23, 2014 (9.28 MB)
  2. 1.0.19 November 26, 2013 (9.28 MB)
  3. 1.0.18 October 17, 2013 (9.28 MB)
  4. 1.0.17 August 23, 2013 (9.28 MB)
  5. 1.0.16 August 22, 2013 (9.28 MB)
Show all versions (82 total)
Ruby Dependency
  1. >= 0