RubyGems Navigation menu

robotstxt 0.5.4

Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Versions:

  1. 0.5.4 - February 12, 2010 (7.5 KB)
  2. 0.5.3 - February 12, 2010 (7.5 KB)
  3. 0.5.2 - December 18, 2009 (7.5 KB)
  4. 0.5.1 - December 5, 2009 (5 KB)
  5. 0.5.0 - December 5, 2009 (5 KB)

Development Dependencies:

Authors:

  • Simone Rinzivillo

Owners:

A53a56103acd8cf3c35f66e87c3b5a68

Total downloads 7,765

For this version 2,586

Required Ruby Version: None

Licenses:

N/A

Gemfile:
gem 'robotstxt', '~> 0.5.4' = Copy to clipboard Copied!

install:
gem install robotstxt = Copy to clipboard Copied!

Links: