RubyGems.org

robotstxt

0.5.4

Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

installgem install robotstxt
Authors

Simone Rinzivillo

6,843 total downloads 2,222 for this version
Owners

A53a56103acd8cf3c35f66e87c3b5a68

Gemfile
gem 'robotstxt', '~> 0.5.4'
Versions
  1. 0.5.4 February 12, 2010 (7.5 KB)
  2. 0.5.3 February 12, 2010 (7.5 KB)
  3. 0.5.2 December 18, 2009 (7.5 KB)
  4. 0.5.1 December 5, 2009 (5 KB)
  5. 0.5.0 December 5, 2009 (5 KB)
Development Dependencies
  1. echoe ~> 3.1
  2. rake ~> 0.8