RubyGems Navigation menu

arachnid 0.2.2

Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.

Gemfile:
= Copy to clipboard Copied!

install:
=

Versions:

  1. 0.4.1 - January 17, 2014 (5 KB)
  2. 0.4.0 - January 16, 2014 (4.5 KB)
  3. 0.3.2 - January 16, 2014 (5 KB)
  4. 0.3.1 - January 16, 2014 (5 KB)
  5. 0.3.0 - January 16, 2014 (4.5 KB)
  6. 0.2.2 - November 29, 2011 (4.5 KB)
Show all versions (12 total)

Runtime Dependencies (4):

bloomfilter-rb = 2.1.1
domainatrix = 0.0.10
nokogiri = 1.5.0
typhoeus = 0.3.2

Owners:

Authors:

  • dchuk

SHA 256 checksum:

c51f575e558abb0b0b43aeeb82bebea45a99e4978e6877c9826aabb94f8216e8

Total downloads 48,522

For this version 4,659

Licenses:

N/A

Required Ruby Version: None

Links: