Make a cache of robots.txt #16
Description
Currently the crawler pings robots.txt everytime, even for repeated URLs. For every hostname, make a cache of allowed and disallowed paths to make the crawling faster. Speed increase: 25-30%
{{ message }}
Currently the crawler pings robots.txt everytime, even for repeated URLs. For every hostname, make a cache of allowed and disallowed paths to make the crawling faster. Speed increase: 25-30%