robotstxt 的反向依赖以下 Gem 的最新版本需要 robotstxt

hostelify 0.8.4

scrapper for publicly accessible data from hostelworld hostelbookers etc.

115,181 下载

site_analyzer 0.3.17

Create site report for SEO many options.

42,043 下载

rwspider 0.4.3

RW Spider is an multithreading spider client written in Ruby designed to make easy th...

8,015 下载