A tool to determine if the robots.txt would prevent a given user agent from making a request to a given URI.

Required Ruby Version

None

Authors

James Edward Gray II, Jeremy Friesen, Edgar Gonzalez

Versions

  1. 0.9.3

    1. source

      1. 0.9.3 source July 21, 2010 (7 KB)
  2. 0.9.2

    1. source

      1. 0.9.2 source July 21, 2010 (7 KB)
  3. 0.9.1

    1. source

      1. 0.9.1 source July 09, 2010 (7 KB)

SHA 256 checksum

← Previous version