When I view my site's robots.txt file in my browser, I notice that the Sitemap URL is incorrect:
http://expats.askbot.com/robots.txt:
User-agent: * Disallow: Sitemap: http://example.com/sitemap.xml
Note that the URL for the sitemap uses example.com as its domain.
Following the instructions in http://askbot.org/en/question/2715/sitemapxml-is-using-examplecom, I changed the domain for my website, and now http://expats.askbot.com/sitemap.xml is displaying the correct URLs, but robots.txt is still broken.
Fixed:
http://expats.askbot.com/robots.txt
User-agent: *
Disallow:
Sitemap: http://expats.askbot.com/sitemap.xml
Thanks! Will this take effect for any domain name (i.e., as specified in Django admin)? Or will it need to be manually updated if/when I change the domain name for the site?
todofixthis ( 2012-03-23 12:53:03 -0500 )editLooks like this might be getting set globally; I accessed <http://expats.askbot.com/robots.txt> just now, and the Sitemap: entry is now http://help.simplecv.org/sitemap.xml. According to GWT, http://askcre.com/sitemap.xml has also appeared in robots.txt.
Create your Q&A site at askbot.com. Managed Askbot hosting at just $15/mo. Dedicated hosting, support contracts, consulting services.
create your Q&A siteAsked: 2012-03-23 09:31:44 -0500
Seen: 111 times
Last updated: Mar 23 '12
Copyright Askbot, 2010-2011. Content on this site is licensed under a Creative Commons Attribution Share Alike 3.0 license.
Thanks for reporting this, will take a look now and fix it.
Fitoria ( 2012-03-23 09:52:47 -0500 )edit