Set robots.txt for askbot.org

asked 2015-04-25 11:48:20 -0500

Nemo_bis's avatar

The current robots.txt allows everything, that doesn't seem safe. Especially as there are frequent error 502, it would probably be best to disallow some dynamic pages (search?) and set a crawl-delay of some seconds.

edit retag flag offensive close merge delete