cobweb 0.0.18
Web Crawler that uses resque background job engine to allow you to cluster your crawl.
Gemfile:
=
Copy to clipboard
Copied!
install:
=
Runtime Dependencies (9):
absolutize
>= 0
addressable
>= 0
awesome_print
>= 0
nokogiri
>= 0
redis
>= 0
resque
>= 0
rspec
>= 0
sinatra
>= 0
thin
>= 0