parolkar-obey_robots_dot_txt 1.0.0

Easy to use extension of Net::HTTP to let you obey robots.txt while crawling/scrapping/mining

Gemfile:
= Copy to clipboard Copied!

install:
=

Versions:

1.0.0 - August 10, 2014 (9 KB)

Development Dependencies (1):

hoe >= 2.3.2

Show all transitive dependencies

Authors:

Abhishek Parolkar

SHA 256 checksum:

0d28b2f645d09e0227ec46489d177cb3bc395cb0637a7764fed5e57334867278