RubyGems Navigation menu

cobweb 1.0.21

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Versions:

  1. 1.0.21 - November 5, 2014 (9.28 MB)
  2. 1.0.20 - August 23, 2014 (9.28 MB)
  3. 1.0.19 - November 26, 2013 (9.28 MB)
  4. 1.0.18 - October 17, 2013 (9.28 MB)
  5. 1.0.17 - August 23, 2013 (9.28 MB)
Show all versions (83 total)

Authors:

  • Stewart McKee

Owners:

8cdb978dffae6ab207dbdeb60d01af2f

Total downloads 88,991

For this version 241

Show all versions (83 total)

Required Ruby Version: >= 0

Licenses:

MIT

Gemfile:
gem 'cobweb', '~> 1.0.21' = Copy to clipboard Copied!

install:
gem install cobweb = Copy to clipboard Copied!

Links: