RubyGems Navigation menu

robotstxt 0.5.4

Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Gemfile
          gem 'robotstxt', '~> 0.5.4'
  
  
  
  
  
  
  

install
          gem install robotstxt
  
  
  
  
  
  
  

Authors:

  • Simone Rinzivillo

Owners:

Total downloads 7,736

For this version 2,573

Required Ruby Version: None

Links:

Versions:

  1. 0.5.4 - February 12, 2010 (7.5 KB)
  2. 0.5.3 - February 12, 2010 (7.5 KB)
  3. 0.5.2 - December 18, 2009 (7.5 KB)
  4. 0.5.1 - December 5, 2009 (5 KB)
  5. 0.5.0 - December 5, 2009 (5 KB)

Development Dependencies: