Set robots.txt for askbot.org
 
   
  
 
 
The current robots.txt allows everything, that doesn't seem safe. Especially as there are frequent error 502, it would probably be best to disallow some dynamic pages (search?) and set a crawl-delay of some seconds.
 
 
To enter a block of code:
 
     
  • enter empty line after your previous text
    •  
  • paste or type the code
    •  
  • select the code and press the  button above
    •  
 
Preview: (hide)
  
 
 
 
 
 
 Nemo_bis's avatar

      3
    
 
 
 
 Nemo_bis 
 
 
    asked 10 years ago 
 
 
 
 
        
 
 
 
Comments
 
 
 
  see more comments