Set robots.txt for askbot.org
The current robots.txt allows everything, that doesn't seem safe. Especially as there are frequent error 502, it would probably be best to disallow some dynamic pages (search?) and set a crawl-delay of some seconds.
Comments