RubyGems Navigation menu

robotstxt-parser 0.1.1

Robotstxt-Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Versions:

  1. 0.1.1 - September 18, 2014 (15 KB)
  2. 0.1.0 - September 18, 2014 (15 KB)

Owners:

Authors:

  • Garen Torikian

SHA 256 checksum:

9c432eff837240b9b776183d498997ae4d4447b90b2eb783126bdb6102a667d0

Total downloads 44,498

For this version 43,415

Gemfile:
= Copy to clipboard Copied!

install:
=

License:

MIT

Required Ruby Version: >= 0

Links: