RubyGems Navigation menu

cobweb 1.0.21

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Gemfile
          gem 'cobweb', '~> 1.0.21'
  
  
  
  
  
  
  

install
          gem install cobweb
  
  
  
  
  
  
  

Licenses:

  • MIT

Total downloads 86,512

For this version 155

Show all versions (83 total)

Required Ruby Version: >= 0

Links:

Versions:

  1. 1.0.21 - November 5, 2014 (9.28 MB)
  2. 1.0.20 - August 23, 2014 (9.28 MB)
  3. 1.0.19 - November 26, 2013 (9.28 MB)
  4. 1.0.18 - October 17, 2013 (9.28 MB)
  5. 1.0.17 - August 23, 2013 (9.28 MB)
Show all versions (83 total)