RubyGems.org

The SimpleCrawler module is a library for crawling web sites. The crawler provides comprehensive data from the page crawled which can be used for page analysis, indexing, accessibility checks etc. Restrictions can be specified to limit crawling of binary files.

installgem install simplecrawler
Authors

Peter Krantz

8,988 total downloads 1,172 for this version
Owners

8bc42d2dd981b4ac5007773d20dff616

Gemfile
gem 'simplecrawler', '~> 0.1.8'
Versions
  1. 0.1.8 November 1, 2011 (6.5 KB)
  2. 0.1.7 May 3, 2009 (9.5 KB)
  3. 0.1.6 November 27, 2008 (6.5 KB)
  4. 0.1.4 September 16, 2008 (6.5 KB)
  5. 0.1.3 January 25, 2008 (6.5 KB)
Show all versions (7 total)
Runtime Dependencies
  1. hpricot >= 0.5