Robotstxt-Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Required Ruby Version

>= 0

Authors

Garen Torikian

Versions

  1. 0.1.1

    1. source

      1. 0.1.1 source September 18, 2014 (15 KB)
  2. 0.1.0

    1. source

      1. 0.1.0 source September 18, 2014 (15 KB)

SHA 256 checksum

← Previous version