Set robots.txt for askbot.org

asked 2015-04-25 11:48:20 -0500

Nemo_bis gravatar image

The current robots.txt allows everything, that doesn't seem safe. Especially as there are frequent error 502, it would probably be best to disallow some dynamic pages (search?) and set a crawl-delay of some seconds.

edit retag flag offensive close merge delete