RubyGems.org

robot_rules

0.9.3

A tool to determine if the robots.txt would prevent a given user agent from making a request to a given URI.

installgem install robot_rules
Download Documentation Subscribe Stats
Authors

James Edward Gray II, Jeremy Friesen, Edgar Gonzalez

1,556 total downloads 584 for this version
Owners

013efad1279cbbf5af325aca3266e501

Links
Homepage
Gemfile
gem "robot_rules", "~> 0.9.3"
Versions
  1. 0.9.3 July 21, 2010
  2. 0.9.2 July 21, 2010
  3. 0.9.1 July 9, 2010
Runtime Dependencies
  1. addressable >= 2.1.2