Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Required Ruby Version

None

Required Rubygems Version

>= 1.2

Authors

Simone Rinzivillo

Versions

  1. 0.5.4

    1. source

      1. 0.5.4 source February 13, 2010 (7.5 KB)
  2. 0.5.3

    1. source

      1. 0.5.3 source February 13, 2010 (7.5 KB)
  3. 0.5.2

    1. source

      1. 0.5.2 source December 19, 2009 (7.5 KB)
  4. 0.5.1

    1. source

      1. 0.5.1 source December 06, 2009 (5 KB)
  5. 0.5.0

    1. source

      1. 0.5.0 source December 06, 2009 (5 KB)

SHA 256 checksum

← Previous version