Ask Your Question
2

robots.txt the nightmare config

asked 2013-02-28 16:02:13 -0500

zulp's avatar

updated 2013-02-28 16:02:41 -0500

[root@ks4000391 ~]# curl ---.com/robots.txt
User-agent: *
Disallow: /


Sitemap: ------.com/sitemap.xml
[root@ks4000391 ~]#

Start google fetcher from webmaster tool :

The page could not be crawled at this time because it is blocked by the most recent robots.txt file Googlebot downloaded. Note that if you recently updated the robots.txt file, it may take up to two days before it's refreshed. You can find more information in the Help Center article about robots.txt.

I have tried all combinations in admin/robots/rule but with no success :( Please help !!

edit retag flag offensive close merge delete

Comments

I agree with you, it's quite an annoyance, we've noticed this on one site only after two weeks of upgrading the app.

Evgeny's avatar Evgeny  ( 2013-02-28 16:36:37 -0500 )edit

2 Answers

Sort by ยป oldest newest most voted
1

answered 2013-02-28 16:25:43 -0500

Evgeny's avatar

The problem is that django-robots app made the default policy "disallow everything".

You can downgrade the app to version 0.8.0 and this issue will go away.

We will need to do something about this robots app - either replace it with a plain-text file or add a more reasonable default.

edit flag offensive delete link more
0

answered 2013-02-28 17:15:17 -0500

coldsystem's avatar

how to downgrade then ? (remove and reninstall?

rm -rf /usr/lib/python2.6/site-packages/django_robots-0.9.1-py2.6.egg*

and reinstall 0.8.0?

any other configuration needed ?

edit flag offensive delete link more

Comments

1

To downgrade, type these two commands:

pip uninstall django-robots

pip install django-robots==0.8.0
Fitoria's avatar Fitoria  ( 2013-02-28 17:27:48 -0500 )edit

Right, and you might need to clear build directory inside your virtual environment, it happens to act as cache on some systems (like on my mac).

Evgeny's avatar Evgeny  ( 2013-02-28 17:41:24 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2013-02-28 16:02:13 -0500

Seen: 414 times

Last updated: Feb 28 '13