robotstxt

0.5.4

Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

installgem install robotstxt

Download Documentation RSS

Authors

Simone Rinzivillo

5,124 total downloads 1,661 for this version

Owners

Links

Homepage Source Code

Gemfile

gem "robotstxt", "~> 0.5.4"

Versions

0.5.4 February 12, 2010
0.5.3 February 12, 2010
0.5.2 December 18, 2009
0.5.1 December 5, 2009
0.5.0 December 5, 2009

Development Dependencies

echoe ~> 3.1
rake ~> 0.8