Set robots.txt for askbot.org

The current robots.txt allows everything, that doesn't seem safe. Especially as there are frequent error 502, it would probably be best to disallow some dynamic pages (search?) and set a crawl-delay of some seconds.

Nemo_bis's avatar
3
Nemo_bis
asked 2015-04-25 11:48:20 -0500
edit flag offensive 0 remove flag close merge delete

Comments

add a comment see more comments