RubyGems.org

arachnid

0.4.1

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

installgem install arachnid
Authors

dchuk

12,634 total downloads 467 for this version
Owners

9ac85e387283cefc5a48e463f812ef71

Gemfile
gem 'arachnid', '~> 0.4.1'
Versions
  1. 0.4.1 January 16, 2014 (5 KB)
  2. 0.4.0 January 16, 2014 (4.5 KB)
  3. 0.3.2 January 16, 2014 (5 KB)
  4. 0.3.1 January 16, 2014 (5 KB)
  5. 0.3.0 January 16, 2014 (4.5 KB)
Show all versions (12 total)